Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-8275

Beam SQL should support BigQuery in DIRECT_READ mode

Details

    • New Feature
    • Status: Resolved
    • P2
    • Resolution: Fixed
    • None
    • 2.16.0
    • dsl-sql
    • None

    Description

      SQL currently only supports reading from BigQuery in DEFAULT (EXPORT) mode. We also need to support DIRECT_READ mode. The method should be configurable by TBLPROPERTIES through the SQL CLI. This will enable us to take advantage of the DIRECT_READ features for filter and project push down.

      References:

      https://beam.apache.org/documentation/io/built-in/google-bigquery/#storage-api

      https://beam.apache.org/blog/2019/06/04/adding-data-sources-to-sql.html

      https://github.com/apache/beam/blob/c2f0d282337f3ae0196a7717712396a5a41fdde1/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTable.java

      Attachments

        Activity

          People

            kirillkozlov Kirill Kozlov
            apilloud Andrew Pilloud
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 2.5h
                2.5h