Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-17204 support un-bucketed tables in acid
  3. HIVE-15899

Make CTAS with acid target table and insert into acid_tbl select ... union all ... work

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 3.0.0
    • Transactions
    • None

    Description

      Consider:

      create table T stored as ORC TBLPROPERTIES('transactional'='true') as
            select a, b from A where a <= 5 union all select a, b from B where a >= 5
      

      and

      create table T (a int, b int) stored as ORC  TBLPROPERTIES ('transactional'='false';
      insert into T(a,b) select a, b from T where a between 1 and 3 group by a, b union all select a, b from A where a between 5 and 7 union all select a, b from B where a >= 9
      

      On Tez, there is an optimization that removes Union All operator writes the data into
      subdirectories of T (in this case T is unpartitioned).

      This also happens on MR but requires

      hiveConf.setBoolVar(HiveConf.ConfVars.HIVE_OPTIMIZE_UNION_REMOVE, true);
      hiveConf.setVar(HiveConf.ConfVars.HIVEFETCHTASKCONVERSION, "none");
      

      Need to ensure that when target table is Acid, we generate unique ROW__IDs
      When target is not acid, that we can convert it to Acid via Alter Table even when data layout includes subdirectories.

      Attachments

        1. HIVE-15899.01.patch
          30 kB
          Eugene Koifman
        2. HIVE-15899.02.patch
          30 kB
          Eugene Koifman
        3. HIVE-15899.03.patch
          30 kB
          Eugene Koifman
        4. HIVE-15899.04.patch
          43 kB
          Eugene Koifman
        5. HIVE-15899.05.patch
          55 kB
          Eugene Koifman
        6. HIVE-15899.07.patch
          68 kB
          Eugene Koifman
        7. HIVE-15899.08.patch
          68 kB
          Eugene Koifman
        8. HIVE-15899.09.patch
          69 kB
          Eugene Koifman
        9. HIVE-15899.10.patch
          72 kB
          Eugene Koifman
        10. HIVE-15899.11.patch
          80 kB
          Eugene Koifman
        11. HIVE-15899.12.patch
          91 kB
          Eugene Koifman
        12. HIVE-15899.13.patch
          91 kB
          Eugene Koifman

        Issue Links

          Activity

            People

              ekoifman Eugene Koifman
              ekoifman Eugene Koifman
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: