Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-14269 Performance optimizations for data on S3
  3. HIVE-14165

Remove Hive file listing during split computation

    XMLWordPrintableJSON

Details

    Description

      The Hive side listing in FetchOperator.java is unnecessary, since Hadoop's FileInputFormat.java will list the files during split computation anyway to determine their size. One way to remove this is to catch the InvalidInputFormat exception thrown by FileInputFormat#getSplits() on the Hive side instead of doing the file listing beforehand.

      For S3 select queries on partitioned tables, this results in a 2x speedup.

      Attachments

        1. HIVE-14165.02.patch
          4 kB
          Abdullah Yousufi
        2. HIVE-14165.03.patch
          4 kB
          Abdullah Yousufi
        3. HIVE-14165.04.patch
          4 kB
          Sahil Takiar
        4. HIVE-14165.05.patch
          4 kB
          Sahil Takiar
        5. HIVE-14165.06.patch
          4 kB
          Sahil Takiar
        6. HIVE-14165.07.patch
          4 kB
          Sahil Takiar
        7. HIVE-14165.patch
          4 kB
          Abdullah Yousufi

        Issue Links

          Activity

            People

              pvargacl Peter Varga
              ayousufi Abdullah Yousufi
              Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 40m
                  40m