Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-8125

Limit number of files generated by insert

    XMLWordPrintableJSON

Details

    Description

      One pitfall of multithreaded execution is that, if implemented naively, the number of files generated by an unpartitioned insert will be multiplied by mt_dop.

      We should provide a mechanism to limit the number of files generated, e.g. limit the number of insert fragment instances (note that there a pre-existing problem with unpartitioned inserts generating too many files).

      Attachments

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-4746 num_nodes should take any value and use that many nodes

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-1108 Impala should check the number of opened files/partition during insert

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          relates to

          New Feature - A new feature of the product, which has yet to be developed. IMPALA-1073 Consider extending the hints for insert

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Activity

            People

              bikramjeet.vig Bikramjeet Vig
              tarmstrong Tim Armstrong
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: