Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-8851

Dataflow BiqQuery TimePartitioning integer overflow

Details

    • Bug
    • Status: Resolved
    • P3
    • Resolution: Invalid
    • 2.16.0
    • 2.16.0
    • io-java-gcp
    • None
    • GCP

    Description

      BigQueryIO.writeTableRows()....withTimePartitioning() seems to fail to create a new BQ table if TimePartitioning expirationMs is bigger than max int. 

      TimePartitioning.setExpirationMs accepts a long and if the expiry time is set to one month (86400 * 1000 * 30 = 2592000000 ms) the BQ server responds with a 400 and the error indicates an integer overflow.

       

      exception: "org.apache.beam.sdk.util.UserCodeException: java.lang.RuntimeException: com.google.api.client.googleapis.json.GoogleJsonResponseException: 400 Bad Request { "code" : 400, "errors" : [

      { "domain" : "global", "message" : "TTL for interval partition must be a positive number.", "reason" : "invalid" }

      ], "message" : "TTL for interval partition must be a positive number.", "status" : "INVALID_ARGUMENT" }

       

      at org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:34) at org.apache.beam.sdk.io.gcp.bigquery.CreateTables$CreateTablesFn$DoFnInvoker.invokeProcessElement(Unknown Source) at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:218)

      Attachments

        Activity

          People

            Unassigned Unassigned
            dev2null Per Böckman
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: