Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-573

TextIO.read() should support sharding

Details

    • Improvement
    • Status: Resolved
    • P2
    • Resolution: Not A Bug
    • 0.1.0-incubating, 0.2.0-incubating
    • Not applicable
    • io-java-text
    • None

    Description

      Currently, TextIO supports sharding (or withoutSharding() option) only in Write.

      I think it makes sense to also supports sharding on the Read behaving like:
      read the input in chunks of n MB (to avoid heap OutOfMemory with large files).

      Attachments

        Activity

          People

            jbonofre Jean-Baptiste Onofré
            jbonofre Jean-Baptiste Onofré
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: