Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-12975

Rework organization of Hadoop dependency on ARM builds

    XMLWordPrintableJSON

Details

    • Task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • Infrastructure
    • None
    • ghx-label-8

    Description

      The hadoop binaries that we download from the CDP build number are built for x86_64. On x86_64, HADOOP_LIB_DIR and HADOOP_INCLUDE_DIR point to the CDP hadoop (i.e. HADOOP_HOME/lib and HADOOP_HOME/include). Various pieces (including the C++ build) use these environment variables to find the native libraries.

      On ARM, we leave those environment variables pointed to that same location. We fix things up by downloading a separate hadoop-client built for ARM, then copying the contents into the usual location in the CDP hadoop directory, overwriting the x86_64 contents. The code to overwrite the libraries runs on each invocation of buildall.sh

      On ARM, we could change this to point HADOOP_LIB_DIR to the downloaded hadoop-client (which is built for ARM). With a bit of work on the hadoop-client, we could get it to also have the header files and also point HADOOP_INCLUDE_DIR to it. This avoids the need to copy files during buildall.sh. Any build that wants to pass in a custom hadoop can then use HADOOP_LIB_DIR_OVERRIDE and HADOOP_INCLUDE_DIR_OVERRIDE.

      Attachments

        Activity

          People

            Unassigned Unassigned
            joemcdonnell Joe McDonnell
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: