Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-27991

Utilise FanoutWriters when inserting records in an Iceberg table when the records are unsorted

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 4.0.0
    • None

    Description

      FanoutWriter is a writer in Iceberg which can be used to write records in a table. This writer keeps all the file handles open, until the write is finished. FanoutWriters is used as the writer when the incoming records are unsorted. We can by default have some mechanism to switch to using FanoutWriters instead of ClusteredWriters when custom sort expressions are not present for the given table/query.
      Similar stuff is already implemented in Spark - https://github.com/apache/iceberg/pull/8621

      Attachments

        Issue Links

          Activity

            People

              sbadhya Sourabh Badhya
              sbadhya Sourabh Badhya
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: