Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-4228

TezClassLoader should be used in TezChild and for Configuration objects

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • None
    • 0.10.0, 0.9.3
    • None

    Description

      While testing a 0.10.0 staging artifact, I found that there multiple issues of TEZ-3860/TEZ-4223, which should be fixed before 0.10.0 release. The issues came up while hive precommit tests:
      http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-1311/11/tests

      The failing hive test uses ADD JAR functionality:

      ADD JAR ${system:maven.local.repository}/org/apache/hive/hive-it-test-serde/${system:hive.version}/hive-it-test-serde-${system:hive.version}.jar;
      

      1. There are code paths, where Configuration object's classloader is used, so if we add resources to TezClassLoader, we should point Configuration objects to use them, otherwise, we'll face exception as this: syslog_bad.log .
      The changes in TezUtils solved the problem for container mode as syslog_good.log

      2. There are codepaths, which use thread context classloader, so calling TezClassLoader.setupTezClassLoader early in TezChild solves issues like this: syslog_bad_context_classloader.log

      3. The Configuration issue is the same in Hive LLAP, which is solved in 1), example failure reproduced locally is: hive_llap.log.

      4. The context classloader problem is also present in LLAP daemons, which is being solved by HIVE-24108

      Attachments

        1. hive_llap.log
          1.03 MB
          László Bodor
        2. syslog_bad_context_classloader.log
          125 kB
          László Bodor
        3. syslog_bad.log
          177 kB
          László Bodor
        4. syslog_good.log
          75 kB
          László Bodor
        5. TEZ-4228.01.patch
          4 kB
          László Bodor

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            abstractdog László Bodor
            abstractdog László Bodor
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment