Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-22476

Hive datediff function provided inconsistent results when hive.fetch.task.conversion is set to none

    XMLWordPrintableJSON

Details

    Description

      The actual issue stems to the different date parser used by various part of the engine.
      Fetch task uses udfdatediff via 

       org.apache.hadoop.hive.ql.udf.generic.GenericUDFToDate

      while the vectorized llap execution uses 

      VectorUDFDateDiffScalarCol

      .

      This fix is meant to be not very intrusive and will add more support to the GenericUDFToDate by enhancing the parser.
      For the longer term will be better to use one parser for all the operators.

      Thanks Rajkumar Singh for the repro example

       
create external table testdatediff(datetimecol string) stored as orc;
insert into testdatediff values ('2019-09-09T10:45:49+02:00'),('2019-07-24');
select datetimecol from testdatediff where datediff(cast(current_timestamp as string), datetimecol)<183;

set hive.ferch.task.conversion=none;
select datetimecol from testdatediff where datediff(cast(current_timestamp as string), datetimecol)<183;

      Attachments

        1. HIVE-22476.10.patch
          8 kB
          Ashutosh Chauhan
        2. HIVE-22476.9.patch
          11 kB
          Slim Bouguerra
        3. HIVE-22476.8.patch
          11 kB
          jcamachorodriguez
        4. HIVE-22476.8.patch
          11 kB
          Slim Bouguerra
        5. HIVE-22476.8.patch
          11 kB
          Slim Bouguerra
        6. HIVE-22476.7.patch
          10 kB
          Slim Bouguerra
        7. HIVE-22476.7.patch
          10 kB
          Slim Bouguerra
        8. HIVE-22476.6.patch
          10 kB
          Slim Bouguerra
        9. HIVE-22476.5.patch
          9 kB
          Slim Bouguerra
        10. HIVE-22476.3.patch
          9 kB
          Slim Bouguerra
        11. HIVE-22476.2.patch
          6 kB
          Slim Bouguerra

        Issue Links

          is duplicated by

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-22459 Hive datadiff function provided inconsistent results when hive.ferch.task.conversion is set to none

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          links to

          Web Link GitHub Pull Request #1046

          Activity

            People

              bslim Slim Bouguerra
              bslim Slim Bouguerra
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m