Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-22360

MultiDelimitSerDe returns wrong results in last column when the loaded file has more columns than those in table schema

    XMLWordPrintableJSON

Details

    Description

      Repro steps:

      Input file:

      1^,1^,^,0^,0^,0 
      2^,1^,^,0^,1^,0 
      3^,1^,^,0^,0^,0 
      4^,1^,^,0^,1^,0
      

      Queries:

      CREATE TABLE  n2(colA int, colB tinyint, colC timestamp, colD smallint, colE smallint) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.MultiDelimitSerDe' WITH SERDEPROPERTIES ("field.delim"="^,")STORED AS TEXTFILE;
      
      LOAD DATA LOCAL INPATH '/Users/schaurasia/Documents/input_6_cols.csv' OVERWRITE INTO TABLE n2;
      
       select * from n2;
      // wrong last column results here.
      +----------+----------+----------+----------+----------+
      | n2.cola  | n2.colb  | n2.colc  | n2.cold  | n2.cole  |
      +----------+----------+----------+----------+----------+
      | 1        | 1        | NULL     | 0        | NULL     |
      | 2        | 1        | NULL     | 0        | NULL     |
      | 3        | 1        | NULL     | 0        | NULL     |
      | 4        | 1        | NULL     | 0        | NULL     |
      +----------+----------+----------+----------+----------+
      

      Cause:
      In multi-serde parsing, the total length calculation here: https://github.com/apache/hive/blob/rel/release-3.1.2/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java#L308 does not take extra fields into account.

      Attachments

        1. HIVE-22360.1.patch
          14 kB
          Shubham Chaurasia
        2. HIVE-22360.2.patch
          20 kB
          Shubham Chaurasia
        3. HIVE-22360.3.patch
          19 kB
          Shubham Chaurasia

        Issue Links

          Activity

            People

              ShubhamChaurasia Shubham Chaurasia
              ShubhamChaurasia Shubham Chaurasia
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 40m
                  40m