Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-21564

Load data into a bucketed table is ignoring partitions specs and loads data into default partition.

    XMLWordPrintableJSON

Details

    Description

      When running below command to load data into bucketed tables it is not loading into specified partition instead loaded into default partition.
      LOAD DATA INPATH '/tmp/files/000000_0' OVERWRITE INTO TABLE call PARTITION(year_partition=2012, month=12);

      SELECT * FROM call WHERE year_partition=2012 AND month=12; --> returns 0 rows.

      CREATE TABLE call( 
      date_time_date date, 
      ssn string, 
      name string, 
      location string) 
      PARTITIONED BY ( 
      year_partition int, 
      month int) 
      CLUSTERED BY ( 
      date_time_date) 
      SORTED BY ( 
      date_time_date ASC) 
      INTO 1 BUCKETS 
      STORED AS ORC;
      

      If set hive.exec.dynamic.partition to false, it fails with below error.

      Error: Error while compiling statement: FAILED: SemanticException 1:18 Dynamic partition is disabled. Either enable it by setting hive.exec.dynamic.partition=true or specify partition column values. Error encountered near token 'month' (state=42000,code=40000)
      

      When we "set hive.strict.checks.bucketing=false;", the load works fine.
      This is a behaviour imposed by HIVE-15148 to avoid incorrectly named data files being loaded to the bucketed tables. In customer use case, if the files are named properly with bucket_id (00000_0, 00000_1 etc), then it is safe to set this flag to false.
      However, current behaviour of loading into default partitions when hive.strict.checks.bucketing=true and partitions specified, was a bug injected by HIVE-19311 where the given query is re-written into a insert query (to handle incorrect file names and Orc versions) but missed to incorporate the partitions specs to it.

      Attachments

        1. HIVE-21564.01.patch
          33 kB
          Sankar Hariappan
        2. HIVE-21564.02.patch
          42 kB
          Sankar Hariappan

        Issue Links

          Activity

            People

              sankarh Sankar Hariappan
              sankarh Sankar Hariappan
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 40m
                  40m