Description
In my k8s enviroment, Interactive session's library on spark 3.0.2, classes in packages are cannot imported.
I will try to use livy binary packages below,
1) 0.7.0: binaries in AWS EMR 6.3.0 (for using scala 2.12)
2) 0.8.0-SNAPSHOT:
https://github.com/jahstreet/incubator-livy
3) spark 3.0.2:
- using client mode: https://github.com/apache/spark/tree/v3.0.2
- using cluster mode: https://github.com/apache/spark/pull/32397
ln jupyter notebook with spark magic, configure is that
%%configure -f {... ... "conf": {"spark.jars.packages": "mypackages" } ... }
The error is raised when importing some class,
import com.mypackage.test
An error aws encounted: <console>:23:error: object mypackage is not a member of package com import com.mypackage.test
However, it is no problem that calling class by name
Class.forName(com.mypackage.test)
It is only occured in livy interactive session.
livy batch, spark-submit and spark-shell have no problem about it.