Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-4175

Consider removing YarnConfiguration where it's possible

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 0.10.0
    • None
    • None

    Description

      A comment in DAGAppmaster made me think that we don't need to rely on YarnConfiguration in all cases, what if it can be replace with base Configuration...

            // TODO Does this really need to be a YarnConfiguration ?
            Configuration conf = new Configuration(new YarnConfiguration());
      

      In hadoop 3.1.3 source, I cannot see that it adds e.g. yarn-site as a resource:

        public YarnConfiguration() {
          super();
        }
        
        public YarnConfiguration(Configuration conf) {
          super(conf);
          if (! (conf instanceof YarnConfiguration)) {
            this.reloadConfiguration();
          }
        }
      

      UPDATE: YarnConfiguration adds yarn-site as a default resource in a static initializer, so it should be used when yarn configs are expected to be present

      in current codebase:

      grep -iRH "new YarnConfiguration" --include="*.java"
      
      tez-plugins/tez-history-parser/src/main/java/org/apache/tez/history/ATSImportTool.java:    YarnConfiguration yarnConf = new YarnConfiguration(conf);
      tez-plugins/tez-aux-services/src/main/java/org/apache/tez/auxservices/ShuffleHandler.java:    super.serviceInit(new YarnConfiguration(conf));
      tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java:    YarnConfiguration yarnConf = new YarnConfiguration(tezConf);
      tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java:    YarnConfiguration yarnConf = new YarnConfiguration(tezConf);
      tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java:    YarnConfiguration yarnConf = new YarnConfiguration(tezConf);
      tez-api/src/test/java/org/apache/tez/client/TestTezClient.java:    tezClient.init(new TezConfiguration(false), new YarnConfiguration());
      tez-api/src/main/java/org/apache/tez/client/TezClient.java:    amConfig.setYarnConfiguration(new YarnConfiguration(amConfig.getTezConfiguration()));
      tez-api/src/main/java/org/apache/tez/client/TezClient.java:    amConfig.setYarnConfiguration(new YarnConfiguration(amConfig.getTezConfiguration()));
      tez-api/src/main/java/org/apache/tez/client/TezClient.java:    return getDAGClient(appId, tezConf, new YarnConfiguration(tezConf), frameworkClient, ugi);
      tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java:      tezConf = new TezConfiguration(new YarnConfiguration());
      tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java:       tezConf = new TezConfiguration(new YarnConfiguration(this.conf));
      tez-mapreduce/src/test/java/org/apache/tez/mapreduce/hadoop/TestMRInputHelpers.java:    Configuration testConf = new YarnConfiguration(
      tez-mapreduce/src/main/java/org/apache/tez/mapreduce/client/YARNRunner.java:   this(conf, new ResourceMgrDelegate(new YarnConfiguration(conf)));
      tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:    Configuration conf = new Configuration(new YarnConfiguration());
      tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:    Configuration conf = new Configuration(new YarnConfiguration());
      tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:    Configuration tezConf = new Configuration(new YarnConfiguration());
      tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:    Configuration tezConf = new Configuration(new YarnConfiguration());
      tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:    Configuration tezConf = new Configuration(new YarnConfiguration());
      tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:    Configuration tezConf = new Configuration(new YarnConfiguration());
      tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:    Configuration tezConf = new Configuration(new YarnConfiguration());
      tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:    Configuration tezConf = new Configuration(new YarnConfiguration());
      tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:    Configuration tezConf = new Configuration(new YarnConfiguration());
      tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:    Configuration tezConf = new Configuration(new YarnConfiguration());
      tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:    Configuration tezConf = new Configuration(new YarnConfiguration());
      tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java:      Configuration conf = new Configuration(new YarnConfiguration());
      

      Attachments

        1. TEZ-4175.01.patch
          28 kB
          László Bodor
        2. TEZ-4175.02.patch
          31 kB
          László Bodor
        3. TEZ-4175.03.patch
          30 kB
          László Bodor
        4. TEZ-4175.03.patch
          30 kB
          László Bodor
        5. TEZ-4175.04.patch
          30 kB
          László Bodor
        6. TEZ-4175.05.patch
          28 kB
          László Bodor

        Issue Links

          Activity

            People

              abstractdog László Bodor
              abstractdog László Bodor
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: