Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-22476

Hive datediff function provided inconsistent results when hive.fetch.task.conversion is set to none

    XMLWordPrintableJSON

Details

    Description

      The actual issue stems to the different date parser used by various part of the engine.
      Fetch task uses udfdatediff via

       org.apache.hadoop.hive.ql.udf.generic.GenericUDFToDate

      while the vectorized llap execution uses

      VectorUDFDateDiffScalarCol

      .

      This fix is meant to be not very intrusive and will add more support to the GenericUDFToDate by enhancing the parser.
      For the longer term will be better to use one parser for all the operators.

      Thanks Rajkumar Singh for the repro example

       
      create external table testdatediff(datetimecol string) stored as orc;
      insert into testdatediff values ('2019-09-09T10:45:49+02:00'),('2019-07-24');
      select datetimecol from testdatediff where datediff(cast(current_timestamp as string), datetimecol)<183;
      
      set hive.ferch.task.conversion=none;
      select datetimecol from testdatediff where datediff(cast(current_timestamp as string), datetimecol)<183;
      

      Attachments

        1. HIVE-22476.10.patch
          8 kB
          Ashutosh Chauhan
        2. HIVE-22476.2.patch
          6 kB
          Slim Bouguerra
        3. HIVE-22476.3.patch
          9 kB
          Slim Bouguerra
        4. HIVE-22476.5.patch
          9 kB
          Slim Bouguerra
        5. HIVE-22476.6.patch
          10 kB
          Slim Bouguerra
        6. HIVE-22476.7.patch
          10 kB
          Slim Bouguerra
        7. HIVE-22476.7.patch
          10 kB
          Slim Bouguerra
        8. HIVE-22476.8.patch
          11 kB
          jcamachorodriguez
        9. HIVE-22476.8.patch
          11 kB
          Slim Bouguerra
        10. HIVE-22476.8.patch
          11 kB
          Slim Bouguerra
        11. HIVE-22476.9.patch
          11 kB
          Slim Bouguerra

        Issue Links

          Activity

            People

              bslim Slim Bouguerra
              bslim Slim Bouguerra
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m