Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-11208

BigQuery storage streams fail with QUOTA_EXCEEDED errors in split

Details

    • Bug
    • Status: Resolved
    • P2
    • Resolution: Fixed
    • 2.25.0
    • 2.27.0
    • io-java-gcp

    Description

      The BigQueryStorageStreamSource attempts to call [SplitReadStream](https://cloud.google.com/bigquery/docs/reference/storage/rpc/google.cloud.bigquery.storage.v1#bigqueryread) for each splitAtFraction call. However, the storage API [limits](https://cloud.google.com/bigquery/quotas#storage-limits) the number of control plane operations per minute for a given project, which can lead to the service being overloaded in the event of a large pipeline. The stream reader should not attempt to split a stream once it has learned that the stream can no longer be split (e.g. once a SplitReadStream call returns an empty response).

      Attachments

        Issue Links

          Activity

            People

              vachan Vachan Shetty
              kjung520 Kenneth Jung
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - 168h
                  168h
                  Remaining:
                  Remaining Estimate - 166.5h
                  166.5h
                  Logged:
                  Remaining Estimate - 166.5h
                  1.5h