Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-21215

Read Parquet INT64 timestamp

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 4.0.0-alpha-1
    • None
    • None

    Description

      This patch enables Hive to start reading timestamps from Parquet written with the new semantics:

      With Parquet version 1.11, a new timestamp LogicalType with base INT64 and the following metadata is introduced:

      • boolean isAdjustedToUtc: marks whether the timestamp is converted to UTC (aka Instant semantics) or not (LocalDateTime semantics).
      • enum TimeUnit (NANOS, MICROS, MILLIS): granularity of timestamp

      Upon reading, the semantics of these new timestamps will be determined by their metadata, while the semantics of INT96 timestamps will continue to be deduced from the writer metadata.
      This feature will be behind a flag for now.

      Attachments

        1. HIVE-21215.1.patch
          18 kB
          Marta Kuczora
        2. HIVE-21215.2.patch
          19 kB
          Marta Kuczora

        Issue Links

          blocks

          New Feature - A new feature of the product, which has yet to be developed. HIVE-21216 Write Parquet INT64 timestamp

          • Major - Major loss of function.
          • Closed
          causes

          Bug - A problem which impairs or prevents the functions of the product. HIVE-26612 INT64 Parquet timestamps cannot be read into BIGINT Hive type

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HIVE-26658 INT64 Parquet timestamps cannot be mapped to most Hive numeric types

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          duplicates

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-13435 Hive fails to read timestamp stored as binary / int64 from externally generated parquet files

          • Major - Major loss of function.
          • Resolved
          is blocked by

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-21050 Use Parquet LogicalTypes

          • Major - Major loss of function.
          • Closed

          Activity

            People

              kuczoram Marta Kuczora
              klcopp Karen Coppage
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: