Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-8933

BigQuery IO should support reading Arrow format over Storage API

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • P3
    • Resolution: Done
    • None
    • Missing
    • io-java-gcp
    • None

    Description

      As of right now BigQuery uses Avro format for reading and writing.

      We should add a config to BigQueryIO to specify which format to use: Arrow or Avro (with Avro as default).

      Attachments

        Issue Links

          blocks

          Improvement - An improvement or enhancement to an existing feature or task. BEAM-9783 Add an Arrow Flight based Java IO connector

          • P3 - Non-urgent bugs, features, and improvements
          • Open
          is a parent of

          Improvement - An improvement or enhancement to an existing feature or task. BEAM-9114 BigQueryIO TableRowParser should support Arrow and Avro data formats

          • P3 - Non-urgent bugs, features, and improvements
          • Open
          links to

          Web Link GitHub Pull Request #10334

          Web Link GitHub Pull Request #10369

          Web Link GitHub Pull Request #10384

          Web Link GitHub Pull Request #10572

          Web Link GitHub Pull Request #14586

          Activity

            People

              miguel.anzo Miguel Anzo
              kirillkozlov Kirill Kozlov
              Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 83h
                  83h