Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-15571

Support Insert into for druid storage handler

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 3.0.0
    • Druid integration
    • None

    Description

      Add support of inset into operator for druid storage handler.

      Implementation -
      1) Using DruidOutputFormat, multiple reducers write segments to a staging directory with the new versions.
      2) When doing insert into, before publishing the segments to the metadata storage, versions for the segments are updated based on the existing segments in druid metadata storage.
      3) After final versions are checked, the segments are moved from the staging directory to the segment directory.
      4) New segment versions are published to the metadata storage.
      Reading of existing segments from metadata storage and updating the versions of newly created segments based on them is done in a single transaction. This will make sure that segments generated by parallel Insert Into statements being executed will not overwrite each other. In case of conflict in metadata storage transaction insert into will fail.

      Attachments

        1. HIVE-15571.01.patch
          26 kB
          Nishant Bangarwa
        2. HIVE-15571.02.patch
          57 kB
          Nishant Bangarwa
        3. HIVE-15571.03.patch
          56 kB
          Nishant Bangarwa

        Issue Links

          Activity

            People

              nishantbangarwa Nishant Bangarwa
              bslim Slim Bouguerra
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: