Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-2741

Python script throws an NameError: name 'Configuration' is not defined in case cache dir is not created

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.10.0
    • 0.11, 0.10.1
    • impl
    • None
    • Pig 0.10

    • Reviewed

    Description

      I have a Python script which writes out data to HDFS

      from org.apache.hadoop.conf import *
      from org.apache.hadoop.fs import *
      
      config = Configuration()
      hdfs = FileSystem.get(config)
      out = hdfs.create(Path("/user/viraj/junk.txt"))
      out.write("Hello World!")
      

      When I run this I get the following error:

      2012-06-06 01:20:43,101 [main] INFO org.apache.pig.Main - Logging error messages to: /home/viraj/pig_1338945643097.log
      2012-06-06 01:20:43,502 [main] INFO org.apache.pig.Main - Run embedded script: jython
      2012-06-06 01:20:43,603 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://namenode:8020
      2012-06-06 01:20:44,069 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: jobtracker:50300
      sys-package-mgr: can't create package cache dir, '/mydir/xx'
      2012-06-06 01:20:45,815 [main] INFO org.apache.pig.scripting.jython.JythonScriptEngine - created tmp python.cachedir=/tmp/pig_jython_7126458276821733512
      2012-06-06 01:20:45,904 [main] ERROR org.apache.pig.Main - ERROR 1121: Python Error. Traceback (most recent call last):
      File "/homes/viraj/test.py", line 4, in <module>
      config = Configuration()
      NameError: name 'Configuration' is not defined

      I tried to solve it in various ways:

      1) Override pig.properties to specify python.cachedir.skip=false but it does not seem to work

      2) The only workaround is to: specify: -Dpython.cachedir=/mydirectory/tmp on the command line

      Viraj

      Attachments

        1. pig-2741-testfailing-pig2665-v6.patch.txt
          3 kB
          Koji Noguchi
        2. pig-2741-testfailing-pig2665-v5.patch.txt
          3 kB
          Koji Noguchi
        3. pig-2741-testfailing-pig2665-v5.patch.txt
          3 kB
          Daniel Dai
        4. pig-2741-testfailing-pig2665-v4.patch.txt
          3 kB
          Daniel Dai
        5. pig-2741-testfailing-pig2665-v3.patch.txt
          3 kB
          Koji Noguchi
        6. pig-2741-testfailing-pig2665-v2.patch.txt
          3 kB
          Koji Noguchi
        7. pig-2741-no-test-yet-v1.patch.txt
          2 kB
          Koji Noguchi

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            knoguchi Koji Noguchi
            viraj Viraj Bhat
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment