Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-17246

Fix shaded client for building Hadoop on Windows

    XMLWordPrintableJSON

Details

    Description

      Currently, the shaded client Yetus personality in Hadoop fails to build on Windows - https://github.com/apache/hadoop/blob/4c04a6768c0cb3d5081cfa5d84ffb389d92f5805/dev-support/bin/hadoop.sh#L541-L615.

      This happens due to the integration test failures in Hadoop client modules - https://github.com/apache/hadoop/tree/4c04a6768c0cb3d5081cfa5d84ffb389d92f5805/hadoop-client-modules/hadoop-client-integration-tests.

      There are several issues that need to be addressed in order to get the integration tests working -

      1. Set the HADOOP_HOME, needed by the Mini DFS and YARN clusters spawned by the integration tests.
      2. Add Hadoop binaries to PATH, so that winutils.exe can be located.
      3. Create a new user with Symlink privilege in the Docker image. This is needed for the proper working of Mini YARN cluster, spawned by the integration tests.
      4. Fix a bug in DFSUtilClient.java that prevents colon ( : ) in the path. The colon is used a delimiter for the PATH variable while specifying multiple paths. However, this isn't a delimiter in the case of Windows and must be handled appropriately.

      Attachments

        1. failsafe-reports-for-trying-to-use-GenericTestUtils.zip
          110 kB
          Gautham Banasandra
        2. image-2023-11-03-17-31-14-990.png
          311 kB
          Gautham Banasandra

        Issue Links

          Activity

            People

              gaurava Gautham Banasandra
              gaurava Gautham Banasandra
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: