Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-24151

MultiDelimitSerDe shifts data if strings contain non-ASCII characters

    XMLWordPrintableJSON

Details

    Description

      HIVE-22360 intended to fix another MultiDelimitSerde problem (with NULL last columns) but introduced a regression: the approach of the fix is pretty much all wrong, as the existing logic that operated on bytes got replaced by regex matcher logic which deals in character positions, rather than byte positions. As some non ASCII characters consist of more than 1 byte, the whole record may get shifted due to this.

      With this ticket I'm going to restore the old logic, and apply the proper fix on that, but keeping (and extending) the test cases added with HIVE-22360 so that we have a solution for both issues.

      Attachments

        Issue Links

          Activity

            People

              szita Ádám Szita
              szita Ádám Szita
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 2h
                  2h