Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-7094

DataflowRunner does not scale when reading gzip file

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 2.11.0
    • Fix Version/s: None
    • Labels:
      None
    • Environment:
      Python on Dataflow

      Description

      Hi,

      I have a pipe that ReadFromText() a 700mb gz file from a GS bucket.

      It then parse json, create BigQuery row, and WriteToBigQuery.

      The pipeline above does not scale. If I specify 2 workers on startup it will scale it down to 1 and the throughput remains the same. The job takes 30 minutes.

       

      What I found is that the exact same pipeline, reading the same but uncompressed 11gb file from the same location scales very well. The job only takes 5 minutes.

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              moander2 Morten Andersen
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: