Uploaded image for project: 'Zeppelin'
  1. Zeppelin
  2. ZEPPELIN-3743

java.lang.RuntimeException: Multiple py4j files found under /usr/lib/spark/python/lib

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.8.0
    • Fix Version/s: None
    • Component/s: pySpark, python-interpreter
    • Labels:
    • Environment:

      AWS Linux running EMR 5.16.0 

      Zeppelin 0.8 is installed using the latest official full bundle.

       

      Description

      The following error is mistakenly reported:

        java.lang.RuntimeException: Multiple py4j files found under /usr/lib/spark/python/lib

      The error happens e.g. during the execution of

      import plotly
      

      The content of /usr/lib/spark/python/lib :

      $ ll  /usr/lib/spark/python/lib
      total 580
      -rw-r--r-- 1 root root  42437 29. Jun 18:11 py4j-0.10.7-src.zip
      -rw-r--r-- 1 root root   1445 29. Jun 18:11 PY4J_LICENSE.txt
      lrwxrwxrwx 1 root root     19 20. Aug 14:01 py4j-src.zip -> py4j-0.10.7-src.zip
      -rw-r--r-- 1 root root 543385 29. Jun 18:11 pyspark.zip
      

      That is, a symlink is treated as a file.

      The line of code containing bug:

      https://github.com/apache/zeppelin/blob/001c621c773db837b4c90bbf7868c94f284247ce/spark/interpreter/src/main/java/org/apache/zeppelin/spark/PythonUtils.java#L74

      If symlinks are resolved to the explicit path to the targeted file, then there will be only one file.

       

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              vak Valery Khamenya
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: