Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-3334 Tez Custom Shuffle Handler
  3. TEZ-3725

Cleanup http connections and other unnecessary fields in DAG Deletion tracker classes.

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.9.0
    • None
    • None

    Description

      JIRA to provides fixes for the following feedback comments from jlowe.

      1. LocalContainerLauncher
      I'm confused why we go through the trouble of serializing the hardcoded value of zero into the aux service protocol buffer, stuff it into the env, then immediately go fetch it back out and extract the integer from the byte buffer. Isn't this a complicated way to say, shufflePort = 0?
      
      2. tezDefaultComponentName only needs to be computed when cleanupDagDataOnComplete is true. Actually may not be needed at all, see related comment for DagDeleteRunnable below.
      
      3. DagDeleteRunnable
      tezDefaultComponentName is unused? I think this transitively means pluginName is unused in DeletionTracker which would simplify it's constructor signature.
      
      4. DeletionTracker
      Nit: addNodeShufflePorts method name being plural implies more than one port can be added but it's only for adding a single node, port pair.
      
      5. AMContainerHelpers
      As Sidd mentioned before, we should avoid the redundant conf key lookup when creating each container launch context.
      
      6. DagDeleteRunnable
      Do we need to do any cleanup on the httpConnection?
      
      7. DeletionTrackerImpl
      What if the submission to the executor throws RejectedExecutionException because the executor was already shutdown and a late dagComplete was invoked?
      

      Attachments

        1. TEZ-3725.002.patch
          18 kB
          Kuhu Shukla
        2. TEZ-3725.001.patch
          12 kB
          Kuhu Shukla

        Activity

          People

            kshukla Kuhu Shukla
            kshukla Kuhu Shukla
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: