Uploaded image for project: 'Zeppelin'
  1. Zeppelin
  2. ZEPPELIN-297

Dependency should be loaded in pypsark

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.6.0
    • 0.5.5
    • Interpreters
    • None

    Description

      dependency loaded with %dep should be added in pyspark

      Exemple:

      //Dataframe csv reader added as dependency

      %dep
      z.reset()
      z.load("com.databricks:spark-csv_2.11:1.2.0")

      // Csv reader can be used in scala
      import org.apache.spark.sql.SQLContext

      val sqlContext = new SQLContext(sc)
      val df = sqlContext.read.format("com.databricks.spark.csv").option("header", "true").load("train.csv")
      z.show(df)

      // But not with pyspark
      %pyspark
      from pyspark.sql import SQLContext
      sqlsc = SQLContext(sc)
      sqlsc.read.format('com.databricks.spark.csv').load('train.csv')

      Py4JJavaError: An error occurred while calling o57.load.
      : java.lang.RuntimeException: Failed to load class for data source: com.databricks.spark.csv

      Attachments

        Activity

          People

            moon Lee Moon Soo
            julien.buret Julien Buret
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: