Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-4137

Input/Output/Processor should merge payload to local conf

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • None
    • None
    • None

    Description

      This patch introduces config merging to various Input and Output processors. As described in https://issues.apache.org/jira/browse/TEZ-4073 , we need to reduce the size of the configuration objects transferred over the wire. There are two improvements we are planning to do regarding to that:

      1. Skip sending default configs and configuration coming from xml files in payload
      2. Send dag, vertex and session configurations in layers instead of sending dag + vertex + session configs all together three times.

      In order to achieve these,

      • We need to expose local config on Task side through TaskContext.
      • Input/Output/Processors must merge the config from user payload to local config in their TaskContext

      Since runtime components did not have access to local config before, tez clients sent all config required at runtime in user payload. After this change, tez clients can reduce their payload size.

      Attachments

        1. TEZ-4137.4.patch
          60 kB
          Mustafa İman
        2. TEZ-4137.4.patch
          59 kB
          Mustafa İman
        3. TEZ-4137.3.patch
          60 kB
          Mustafa İman
        4. TEZ-4137.2.patch
          58 kB
          Mustafa İman
        5. TEZ-4137.1.patch
          56 kB
          Mustafa İman

        Issue Links

          Activity

            People

              mustafaiman Mustafa İman
              mustafaiman Mustafa İman
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 0.5h
                  0.5h