Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-3649

HadoopSeekableByteChannel breaks when backing InputStream doesn't supporte ByteBuffers

Details

    Description

      This happened last summer, when I wanted to use S3A as the backing HDFS access implementation. 

      This is because while this method is called : https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataInputStream.java#L145

      This class does not implement ByteBuffer readable https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java

      I fixed it by manually incrementing the read position and copying the backing array instead of buffering.

      https://github.com/Igosuki/beam/commit/3838f0db43b6422833a045d1f097f6d7643219f1

      I know the s3 direct implementation is the preferred path, but this is possible, and likely happens to a lot of developers.

      Attachments

        Activity

          People

            Unassigned Unassigned
            igosuki Guillaume Balaine
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: