Uploaded image for project: 'Tajo (Retired)'
  1. Tajo (Retired)
  2. TAJO-2063

Refactor FileTablespace::commitOutputData.

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Won't Fix
    • None
    • None
    • Storage
    • None

    Description

      FileTablespace::commitOutputData has some problems as following:

      First, it is too long and complexed because it handles various cases in a single method. We need to refactor this method into several small and well-defined methods.

      Second, FileSystem::listStatus is widely used while committing output data. It especially causes a lot of overhead to list partitioned directories in S3. Also it occurs in HDFS too with large partitioned tables. We need to minimize its usage.

      Attachments

        Issue Links

          is depended upon by

          Sub-task - The sub-task of the issue TAJO-1905 Insert clause to partitioned table fails on S3

          • Major - Major loss of function.
          • In Progress

          Sub-task - The sub-task of the issue TAJO-2069 Implement finding the total size of all objects in a bucket with AWS SDK.

          • Major - Major loss of function.
          • Resolved
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. TAJO-2023 Eliminate the use of FileSystem::getContentSummary

          • Major - Major loss of function.
          • Open

          Activity

            People

              blrunner JaeHwa Jung
              blrunner JaeHwa Jung
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: