Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-4228

TezClassLoader should be used in TezChild and for Configuration objects

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • None
    • 0.10.0, 0.9.3
    • None

    Description

      While testing a 0.10.0 staging artifact, I found that there multiple issues of TEZ-3860/TEZ-4223, which should be fixed before 0.10.0 release. The issues came up while hive precommit tests:
      http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-1311/11/tests

      The failing hive test uses ADD JAR functionality:

      ADD JAR ${system:maven.local.repository}/org/apache/hive/hive-it-test-serde/${system:hive.version}/hive-it-test-serde-${system:hive.version}.jar;
      

      1. There are code paths, where Configuration object's classloader is used, so if we add resources to TezClassLoader, we should point Configuration objects to use them, otherwise, we'll face exception as this: syslog_bad.log .
      The changes in TezUtils solved the problem for container mode as syslog_good.log

      2. There are codepaths, which use thread context classloader, so calling TezClassLoader.setupTezClassLoader early in TezChild solves issues like this: syslog_bad_context_classloader.log

      3. The Configuration issue is the same in Hive LLAP, which is solved in 1), example failure reproduced locally is: hive_llap.log.

      4. The context classloader problem is also present in LLAP daemons, which is being solved by HIVE-24108

      Attachments

        1. syslog_bad_context_classloader.log
          125 kB
          László Bodor
        2. TEZ-4228.01.patch
          4 kB
          László Bodor
        3. syslog_good.log
          75 kB
          László Bodor
        4. hive_llap.log
          1.03 MB
          László Bodor
        5. syslog_bad.log
          177 kB
          László Bodor

        Issue Links

          Activity

            People

              abstractdog László Bodor
              abstractdog László Bodor
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: