Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-4192

Attach task specific info in scheduler and retrieve in communicator

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Patch Available
    • Major
    • Resolution: Unresolved
    • 0.9.2
    • None
    • None
    • None

    Description

      There is no way to pass information from scheduler to communicator. Sometimes we want to let communicator know about extra properties of the task other than its spec. One example is if a task is preemptable.

      When scheduler receives a task, its taskspec is already constructed. It is a nice immutable object at this point. We do not want to modify and put more info in it at that stage. So we pass a new piece of information to TaskSchedulerContext#taskAllocated along with the allocation. This info is passed to the communicator untouched. It is up to the communicator implementation to make use of it. Communicator implementation needs to override a new registerRunningTaskAttempt that accepts this additional object along with all the other arguments in the original registerRunningTaskAttempt method.

      The new registerRunningTaskAttempt method has a default implementation that falls back to the old method. So if the user does not override the new method, the same old behavior is preserved. Similarly on scheduler side, the user needs to call new taskAllocated method to pass the additional info object. If they use the old taskAllocated method, old behavior is preserved.

       

      Attachments

        1. TEZ-4192.2.patch
          70 kB
          Mustafa İman
        2. TEZ-4192.1.patch
          49 kB
          Mustafa İman

        Issue Links

          Activity

            People

              mustafaiman Mustafa İman
              mustafaiman Mustafa İman
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m