Uploaded image for project: 'Livy'
  1. Livy
  2. LIVY-857

In k8s, Interactive session's library on spark 3.0.2, classes in packages cannot be imported

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 0.7.0, 0.8.0
    • 0.9.0
    • RSC
    • None

    Description

      In my k8s enviroment, Interactive session's library on spark 3.0.2, classes in packages are cannot imported.

       

      I will try to use livy binary packages below,

      1) 0.7.0: binaries in AWS EMR 6.3.0 (for using scala 2.12)

      2) 0.8.0-SNAPSHOT: 

      https://github.com/jahstreet/incubator-livy

      3) spark 3.0.2: 

       

      ln jupyter notebook with spark magic, configure is that

      %%configure -f
      {...
       ...
       "conf": {"spark.jars.packages": "mypackages" }
       ...
      }

       

      The error is raised when importing some class,

      import com.mypackage.test
      An error aws encounted:
      <console>:23:error: object mypackage is not a member of package com
          import com.mypackage.test

       

      However, it is no problem that calling class by name 

      Class.forName(com.mypackage.test)

       

      It is only occured in livy interactive session.

      livy batch, spark-submit and spark-shell  have no problem about it.

      Attachments

        Activity

          People

            Unassigned Unassigned
            ocworld Keunhyun Oh
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: