Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-12435

Generalize S3FileSystem

Details

    • Improvement
    • Status: Triage Needed
    • P2
    • Resolution: Fixed
    • None
    • 2.32.0
    • io-java-aws

    Description

      I'm working with multiple storage systems that speak the S3 api. I would like to support FileIO operations for these storage systems, but S3FileSystem hardcodes the s3 scheme (the various systems use different URI schemes) and it is in any case impossible to instantiate more than one in the current design.

      I'd like to refactor the code in org.apache.beam.sdk.io.aws.s3 (and maybe ...aws.options) somewhat to enable this use-case. I haven't worked out the details yet, but it will take some thought to make this work in a non-hacky way.

      Attachments

        Issue Links

          Activity

            People

              mrudary Matt Rudary
              mrudary Matt Rudary
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 6h 40m
                  6h 40m