Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
0.6.0
-
None
Description
dependency loaded with %dep should be added in pyspark
Exemple:
//Dataframe csv reader added as dependency
%dep
z.reset()
z.load("com.databricks:spark-csv_2.11:1.2.0")
// Csv reader can be used in scala
import org.apache.spark.sql.SQLContext
val sqlContext = new SQLContext(sc)
val df = sqlContext.read.format("com.databricks.spark.csv").option("header", "true").load("train.csv")
z.show(df)
// But not with pyspark
%pyspark
from pyspark.sql import SQLContext
sqlsc = SQLContext(sc)
sqlsc.read.format('com.databricks.spark.csv').load('train.csv')
Py4JJavaError: An error occurred while calling o57.load.
: java.lang.RuntimeException: Failed to load class for data source: com.databricks.spark.csv