Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-3810

TezCounter for idle time in shuffle phase

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Patch Available
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None
    • None

    Description

      A task attempt counter that tracks how much time was spent waiting for inputs in the shuffle phase. We can use this to quickly identify jobs that are wasting a lot of time on the grid with idle reducer tasks instead of shuffling/merging.

      Attachments

        1. TEZ-3810-001.patch
          6 kB
          Ashwin Ramesh
        2. TEZ-3810.002.patch
          13 kB
          Kuhu Shukla
        3. TEZ-3810.003.patch
          13 kB
          Kuhu Shukla
        4. TEZ-3810.004.patch
          13 kB
          Kuhu Shukla
        5. TEZ-3810.005.patch
          14 kB
          Kuhu Shukla

        Issue Links

          Activity

            People

              kshukla Kuhu Shukla
              aramesh2 Ashwin Ramesh
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated: