Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-7853

Add support to read int64 NANO timestamps to the parquet scanner

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Implemented
    • None
    • Impala 3.2.0
    • Backend

    Description

      PARQUET-1387 added int64 timestamps with nanosecond precision.

      As 64 bits are not enough to represent the whole 1400..9999 range of Impala timestamps, this new new type works with a limited range:
      1677-09-21 00:12:43.145224192 .. 2262-04-11 23:47:16.854775807 UTC

      The benefit of the reduced range is that no validation is necessary during scanning, as every possible 64 bit value represents a valid timestamp in Impala. This may mean that this has the potential be the fastest way to store timestamps in Impala + Parquet.

      Another way NANO differs from MICRO and MILLI is that NANO can be only described with new logical types in Parquet, it has no converted type equivalent.

      Attachments

        Activity

          People

            csringhofer Csaba Ringhofer
            csringhofer Csaba Ringhofer
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: