Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-8495

Change byte size estimate for compressed files for FileBasedSources

Details

    • Bug
    • Status: Open
    • P3
    • Resolution: Unresolved
    • None
    • None
    • io-java-files, io-py-files
    • None

    Description

      Consider change the estimated byte size for file based sources for better initial autoscaling (on runners which support it; currently only Dataflow). Currently the estimate is based on total size, for compressed file we could add a factor multiplier (~5) for compressed files in a blob go produce a better estimate.

      Attachments

        Activity

          People

            Unassigned Unassigned
            altay Ahmet Altay
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated: