Uploaded image for project: 'Zeppelin'
  1. Zeppelin
  2. ZEPPELIN-4175

case class fails on remote spark

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 0.8.0
    • None
    • Interpreters
    • None
    • HDP: 3.0.1

        zeppelin: 0.8.0

        Spark: 2.3.1 

    Description

      I am testing use of scala case class in a zeppelin instance provided by an Horontoworks Data Platform stack.

      The bug is reproducible with following instructions:

       

      Copy/paste in zeppelin instructions from https://zeppelin.apache.org/docs/0.5.5-incubating/tutorial/tutorial.html. (full screenshot attached)

      In a first cell, instructions to create the RDD and then the temporary table.

       

      After execution, I get "console>:37: error: value toDF is not a member of org.apache.spark.rdd.RDD[Bank] bank.toDF().registerTempTable("bank") 

       

      Second cell is the first sql query in the tutorial.

       

      Execution fails with:

      java.lang.ArrayIndexOutOfBoundsException: 5

      at $line541920741114.$read$$iw$$iw$$anonfun$3.apply(<console>:33)

      at $line541920741114.$read$$iw$$iw$$anonfun$3.apply(<console>:29)

      at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)

      at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)

      ...

       

      The exact basic tutorial code (i.e. access to amazone and building a local in memory rdd) seems to work in remote zeppelin (i.e. sql provides results) but I still get  "value toDF is not a member of org.apache.spark.rdd.RDD[Bank] ".

       

      Testing the basic tutorial code in a local 0.8.0 zeppelin works without any issue.

       

      Using code for building an in memory 

      Attachments

        Activity

          People

            Unassigned Unassigned
            gholler Guillaume HOLLER
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: