Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-9995

ACID compaction tries to compact a single file

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 1.0.0
    • 4.0.0-alpha-1
    • Transactions
    • None

    Description

      Consider TestWorker.minorWithOpenInMiddle()

      since there is an open txnId=23, this doesn't have any meaningful minor compaction work to do. The system still tries to compact a single delta file for 21-22 id range, and effectively copies the file onto itself.

      This is 1. inefficient and 2. can potentially affect a reader.

      (from a real cluster)
      Suppose we start with 

      drwxr-xr-x   - ekoifman staff          0 2016-06-09 16:03 /user/hive/warehouse/t/base_0000016
-rw-r--r--   1 ekoifman staff        602 2016-06-09 16:03 /user/hive/warehouse/t/base_0000016/bucket_00000
drwxr-xr-x   - ekoifman staff          0 2016-06-09 16:07 /user/hive/warehouse/t/base_0000017
-rw-r--r--   1 ekoifman staff        588 2016-06-09 16:07 /user/hive/warehouse/t/base_0000017/bucket_00000
drwxr-xr-x   - ekoifman staff          0 2016-06-09 16:07 /user/hive/warehouse/t/delta_0000017_0000017_0000
-rw-r--r--   1 ekoifman staff        514 2016-06-09 16:06 /user/hive/warehouse/t/delta_0000017_0000017_0000/bucket_00000
drwxr-xr-x   - ekoifman staff          0 2016-06-09 16:07 /user/hive/warehouse/t/delta_0000018_0000018_0000
-rw-r--r--   1 ekoifman staff        612 2016-06-09 16:07 /user/hive/warehouse/t/delta_0000018_0000018_0000/bucket_00000

      then do alter table T compact 'minor';

      then we end up with 

      drwxr-xr-x   - ekoifman staff          0 2016-06-09 16:07 /user/hive/warehouse/t/base_0000017
-rw-r--r--   1 ekoifman staff        588 2016-06-09 16:07 /user/hive/warehouse/t/base_0000017/bucket_00000
drwxr-xr-x   - ekoifman staff          0 2016-06-09 16:11 /user/hive/warehouse/t/delta_0000018_0000018
-rw-r--r--   1 ekoifman staff        500 2016-06-09 16:11 /user/hive/warehouse/t/delta_0000018_0000018/bucket_00000
drwxr-xr-x   - ekoifman staff          0 2016-06-09 16:07 /user/hive/warehouse/t/delta_0000018_0000018_0000
-rw-r--r--   1 ekoifman staff        612 2016-06-09 16:07 /user/hive/warehouse/t/delta_0000018_0000018_0000/bucket_00000

      So compaction created a new dir /user/hive/warehouse/t/delta_0000018_0000018

      Attachments

        1. HIVE-9995.WIP.patch
          3 kB
          Eugene Koifman
        2. HIVE-9995.10.patch
          22 kB
          Denys Kuzmenko
        3. HIVE-9995.09.patch
          22 kB
          Denys Kuzmenko
        4. HIVE-9995.08.patch
          14 kB
          Denys Kuzmenko
        5. HIVE-9995.07.patch
          14 kB
          Denys Kuzmenko
        6. HIVE-9995.06.patch
          16 kB
          Denys Kuzmenko
        7. HIVE-9995.05.patch
          16 kB
          Denys Kuzmenko
        8. HIVE-9995.04.patch
          14 kB
          Denys Kuzmenko
        9. HIVE-9995.03.patch
          10 kB
          Denys Kuzmenko
        10. HIVE-9995.02.patch
          10 kB
          Denys Kuzmenko
        11. HIVE-9995.01.patch
          9 kB
          Eugene Koifman

        Issue Links

          fixes

          Bug - A problem which impairs or prevents the functions of the product. HIVE-20901 running compactor when there is nothing to do produces duplicate data

          • Major - Major loss of function.
          • Closed
          is related to

          Sub-task - The sub-task of the issue HIVE-21266 Don't run cleaner if compaction is skipped (issue with single delta file)

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HIVE-16669 Fine tune Compaction to take advantage of Acid 2.0

          • Critical - Crashes, loss of data, severe memory leak.
          • Open
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HIVE-20901 running compactor when there is nothing to do produces duplicate data

          • Major - Major loss of function.
          • Closed

          Activity

            People

              dkuzmenko Denys Kuzmenko
              ekoifman Eugene Koifman
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: