Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-21279

Avoid moving/rename operation in FileSink op for SELECT queries

    XMLWordPrintableJSON

Details

    Description

      Currently at the end of a job FileSink operator moves/rename temp directory to another directory from which FetchTask fetches result. This is done to avoid fetching potential partial/invalid files by failed/runway tasks. This operation is expensive for cloud storage. It could be avoided if FetchTask is passed on set of files to read from instead of whole directory.

      Attachments

        1. HIVE-21279.1.patch
          31 kB
          Vineet Garg
        2. HIVE-21279.10.patch
          31 kB
          Vineet Garg
        3. HIVE-21279.11.patch
          33 kB
          Vineet Garg
        4. HIVE-21279.12.patch
          33 kB
          Vineet Garg
        5. HIVE-21279.13.patch
          33 kB
          Vineet Garg
        6. HIVE-21279.2.patch
          27 kB
          Vineet Garg
        7. HIVE-21279.3.patch
          27 kB
          Vineet Garg
        8. HIVE-21279.4.patch
          27 kB
          Vineet Garg
        9. HIVE-21279.5.patch
          27 kB
          Vineet Garg
        10. HIVE-21279.6.patch
          27 kB
          Vineet Garg
        11. HIVE-21279.7.patch
          27 kB
          Vineet Garg
        12. HIVE-21279.8.patch
          28 kB
          Vineet Garg
        13. HIVE-21279.9.patch
          30 kB
          Vineet Garg

        Issue Links

          Activity

            People

              vgarg Vineet Garg
              vgarg Vineet Garg
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 2h 50m
                  2h 50m