Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-23442

ACID major compaction doesn't read base directory correctly if it was written by insert overwrite

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 4.0.0
    • 4.0.0-alpha-1
    • None
    • None

    Description

      Steps to reproduce:

      SET hive.acid.direct.insert.enabled=true;
      
      CREATE EXTERNAL TABLE test_comp_txt(a int, b int, c int) STORED AS TEXTFILE;
      INSERT INTO test_comp_txt values (1, 1, 1), (2, 2, 2), (3, 3, 3), (4, 4, 4);
      
      CREATE TABLE test_comp(a int, b int, c int) STORED AS ORC TBLPROPERTIES('transactional'='true');
      INSERT OVERWRITE TABLE test_comp SELECT * FROM test_comp_txt;
      
      UPDATE test_comp SET b=55, c=66 WHERE a=2;
      DELETE FROM test_comp WHERE a=4;
      UPDATE test_comp SET b=77 WHERE a=1;
      
      SELECT * FROM test_comp;
      3	3	3
      2	55	66
      1	77	1
      
      ALTER TABLE test_comp COMPACT 'MAJOR';
      
      SELECT * FROM test_comp;
      2	55	66
      1	77	1
      

      This issue only occurs if the base directory was created with an insert overwrite command and the hive.acid.direct.insert.enabled parameter was true. This issue doesn't affect the query based compaction.

      Attachments

        1. HIVE-23442.1.patch
          6 kB
          Marta Kuczora
        2. HIVE-23442.2.patch
          6 kB
          Marta Kuczora

        Issue Links

          Activity

            People

              kuczoram Marta Kuczora
              kuczoram Marta Kuczora
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: