Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-8393

Java BigQueryIO clustering support breaks on multiple partitions

Details

    • Bug
    • Status: Resolved
    • P1
    • Resolution: Fixed
    • 2.15.0, 2.16.0
    • 2.17.0
    • io-java-gcp
    • None

    Description

      Support for writing to clustered tables in BigQuery was added in 2.15, which involved adding a new optional clustering field to TableDestination. Clustering support is working for most cases, but fails with errors about incompatible partitioning specifications for any data that is handled by the MultiplePartitions branch of BigQueryIO logic.

      There is a case in that code path where we provide a modified TableDestination and neglect to copy the clustering definition, so the final load job does not include any clustering columns.

      Attachments

        Issue Links

          Activity

            People

              jeff.klukas@gmail.com Jeff Klukas
              jeff.klukas@gmail.com Jeff Klukas
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 10m
                  1h 10m