Uploaded image for project: 'Zeppelin'
  1. Zeppelin
  2. ZEPPELIN-2589

Let Livy interpreter show traceback log

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.7.1
    • 0.8.0
    • livy-interpreter
    • None
    • zeppelin-0.7.1,livy-0.4-snapshot

    Description

      I am using Livy interpreter to interact with spark cluster. However, every time some error occurs,Zeppelin only print one line of error message which is extremely unhelpful to debug the code.
      For example, If I get the following error output in Zeppelin:

      <console>:113: error: overloaded method value createDataFrame with alternatives:

      I can't found out exactly what caused the error.

      But executing the same code on spark shell I would get the following error :

      <console>:56: error: overloaded method value createDataFrame with alternatives:
      (data: java.util.List[_],beanClass: Class[_])org.apache.spark.sql.DataFrame <and>
      (rdd: org.apache.spark.api.java.JavaRDD[_],beanClass: Class[_])org.apache.spark.sql.DataFrame <and>
      (rdd: org.apache.spark.rdd.RDD[_],beanClass: Class[_])org.apache.spark.sql.DataFrame <and>
      (rows: java.util.List[org.apache.spark.sql.Row],schema: org.apache.spark.sql.types.StructType)org.apache.spark.sql.DataFrame <and>
      (rowRDD: org.apache.spark.api.java.JavaRDD[org.apache.spark.sql.Row],schema: org.apache.spark.sql.types.StructType)org.apache.spark.sql.DataFrame <and>
      (rowRDD: org.apache.spark.rdd.RDD[org.apache.spark.sql.Row],schema: org.apache.spark.sql.types.StructType)org.apache.spark.sql.DataFrame
      cannot be applied to (org.apache.spark.rdd.RDD[String], org.apache.spark.sql.types.StructType)
      val testDF = spark.createDataFrame(rdds, schema)

      All the detailed error log is returned by Livy server, but in another field called "traceback". In BaseLivyInterpreter.java, Zepplein only print out the "evalue" which contain only the header line of the error log.

      I think print "traceback" log would be much more helpful to debug spark job when using Livy.

      Attachments

        Issue Links

          Activity

            People

              why heyang wang
              why heyang wang
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: