Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-4253

Revert TEZ-4170

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.10.0
    • 0.10.1
    • None
    • None

    Description

      There are two performance improvements in https://issues.apache.org/jira/browse/TEZ-4170

      1. Move construction of InputInitializers to background thread
      2. Remove RootInputInitializerManager's thread pool and move all threads using this executor to DagAppMaster's thread pool.

      Item 1: This is an incorrect optimization which may cause data races in VertexImpl regarding handling of events. This was mitigated in https://issues.apache.org/jira/browse/TEZ-4204 however the solution basically reverts back the initial optimization only with a more complicated approach. Apart from these, it unnecessarily complicates Tez application master. The scenario where this is useful is when a custom InputInitializer constructor contains a lot of heavyweight operations. However, the solution to this problem belongs to client application. Client application can easily move heavyweight operations to InputInitializer#initialize method.

      Item 2: The benefit of this is doubtful. Supposedly a root input initializer is offloaded to a cached thread in appcontext instead of creating a new one in RootInputInitializerManager. The number of threads of this pool is limited. When many root input initializer depends on InputInitializerEvents all threads may get blocked. In that case the rest of the vertices (which are supposed to send the InputInitializerEvents) cannot run. So we run into a deadlock.

       

      Attachments

        1. TEZ-4253.patch
          36 kB
          Mustafa İman
        2. TEZ-4253.2.patch
          37 kB
          Mustafa İman
        3. TEZ-4253.2.patch
          37 kB
          Mustafa İman
        4. TEZ-4253.2.patch
          39 kB
          Mustafa İman

        Activity

          People

            mustafaiman Mustafa İman
            mustafaiman Mustafa İman
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 20m
                20m