Uploaded image for project: 'Tajo (Retired)'
  1. Tajo (Retired)
  2. TAJO-931

Output file can be punctuated depending on the file size.

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 0.9.0
    • Physical Operator
    • None

    Description

      There are some file formats (e.g., Parquet) which are not splittable. They can usually span multiple HDFS blocks if one file is very large. It causes remote HDFS access and limits the parallel degree, resulting in significant performance degradation.

      We can solve this problem if StoreTableExec or

      {Col|SortBased}

      PartitionStoreExec can punctuate the final output file according to the written size.

      In addition, we need to support a session variable to determine the per file size of final output files. So, TAJO-928 blocks this issue.

      Attachments

        Issue Links

          blocks

          Improvement - An improvement or enhancement to an existing feature or task. TAJO-935 (ColPartition)StoreTableExec should use per-file block size config for file creations.

          • Major - Major loss of function.
          • Open
          is blocked by

          Improvement - An improvement or enhancement to an existing feature or task. TAJO-928 Session variables should override query configs in TajoConf.

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. TAJO-933 Fork some classes of Parquet as builtin third-party classes

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Activity

            People

              hyunsik Hyunsik Choi
              hyunsik Hyunsik Choi
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: