Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5451

MR uses LD_LIBRARY_PATH which doesn't mean anything in Windows

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.1.1-beta, 3.0.0-alpha1
    • 2.3.0
    • None
    • None
    • Windows

    • Reviewed

    Description

      In order to set the path for loading native libraries, MR relies on the default value of the mapreduce.admin.user.env configuration setting the LD_LIBRARY_PATH environment entry. There are two problems with this setting in Windows:
      a) LD_LIBRARY_PATH doesn't mean anything in Windows.
      b) It sets it using $HADOOP_COMMON_HOME, instead of %HADOOP_COMMON_HOME%.

      The default value here should be platform-dependent (use the PATH variable in Windows instead of LD_LIBRARY_PATH), or we should rely on another mechanism. The net effect is that in Windows unless this configuration is over-ridden MR jobs fail with this error:

      2013-05-29 13:51:41,049 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
      	at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method)
      	at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:393)
      	at org.apache.hadoop.fs.FileUtil.canRead(FileUtil.java:928)
      	at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:177)
      	at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:164)
      	at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:98)
      	at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:288)
      	at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:431)
      	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:164)
      	at org.apache.hadoop.mapred.YarnChild.configureLocalDirs(YarnChild.java:235)
      	at org.apache.hadoop.mapred.YarnChild.configureTask(YarnChild.java:294)
      	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:143)
      

      Attachments

        1. MAPREDUCE-5451.patch
          5 kB
          Yingda Chen
        2. MAPREDUCE-5451-2.patch
          4 kB
          Yingda Chen
        3. MAPREDUCE-5451-3.patch
          4 kB
          Yingda Chen
        4. MAPREDUCE-5451-4.patch
          5 kB
          Chris Nauroth
        5. MAPREDUCE-5451-5.patch
          8 kB
          Yingda Chen
        6. MAPREDUCE-5451-6.patch
          8 kB
          Yingda Chen
        7. MAPREDUCE-5451-7.patch
          8 kB
          Yingda Chen
        8. YARN-729.patch
          5 kB
          Yingda Chen

        Issue Links

          Activity

            People

              yingdachen Yingda Chen
              mostafae Mostafa Elhemali
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: