Uploaded image for project: 'Zeppelin'
  1. Zeppelin
  2. ZEPPELIN-1347 Release 0.6.2
  3. ZEPPELIN-1442

UDF can not be found due to 2 instances of SparkSession is created

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.6.1
    • 0.6.2, 0.7.0
    • pySpark
    • None

    Description

      It worked in 0.5.6 with spark 1.6.2
      On 2.0 - I've checked - when I run just pySpark everything working fine.

      I've build zeppelin with

      mvn clean package -Pspark-2.0 -Phadoop-2.6 -Pyarn -Ppyspark -DskipTests
      
      %pyspark
      from pyspark.sql import SQLContext, Row
      from pyspark.sql.types import StringType, IntegerType, StructType, StructField, MapType, FloatType, ArrayType
      from pyspark.sql.functions import udf
      
      sqlContext.registerFunction("stringLengthString", lambda x: len(x))
      

      Returns:

      Traceback (most recent call last):
        File "/tmp/zeppelin_pyspark-3201872850141735060.py", line 266, in <module>
          raise Exception(traceback.format_exc())
      Exception: Traceback (most recent call last):
        File "/tmp/zeppelin_pyspark-3201872850141735060.py", line 264, in <module>
          exec(code)
        File "<stdin>", line 7, in <module>
        File "/home/egor/spark/python/pyspark/sql/context.py", line 203, in registerFunction
          self.sparkSession.catalog.registerFunction(name, f, returnType)
      AttributeError: 'JavaMember' object has no attribute 'registerFunction'
      

      Attachments

        Issue Links

          Activity

            People

              zjffdu Jeff Zhang
              epahomov Egor Pahomov
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: