Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-9051

BigQueryUtils toBeamRow to support GenericRecords with millisecond precision timestamps

Details

    • Improvement
    • Status: Resolved
    • P3
    • Resolution: Won't Fix
    • 2.16.0
    • Not applicable
    • io-java-gcp
    • None

    Description

      Currently, BigQueryUtils assumes all Avro GenericRecords which utilize timestamp fields include timestamp data in microseconds when converting from GenericRecords to Beam Rows (https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java#L568). The implementation should also support timestamp data in milliseconds; this should be feasible by referencing the LogicalType associated with the avro field to determine if the underlying value is represented in millis or micros.

      Attachments

        Issue Links

          Activity

            People

              Ryan Berti Ryan Berti
              Ryan Berti Ryan Berti
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 2h
                  2h