Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-4479

Eagerly Init/Load FileSystem In Tez Task Containers

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • 0.10.4
    • None
    • None

    Description

      Initing/Loading FileSystem such as S3 can take ~10s - ~20s when called for the first time and the time taken for subsequent calls are negligable. If we can load the FileSystem much before it is used can help us to save some time. It can be especially useful in case of pre-warm Tez containers where the Tez task containers comes up when the Application Master (AM) is launched and not on-demand which is the default behavior. It can be also useful in cases where the Mapper tasks spends considerable time consuming the upstream shuffle data and then heads to process some FileSystem operations, in all such cases we have few FileSystem load up time.

      Attachments

        Issue Links

          Activity

            People

              srahman Syed Shameerur Rahman
              srahman Syed Shameerur Rahman
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 2h 40m
                  2h 40m