Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-4861

Hadoop Filesystem silently fails

Details

    • Bug
    • Status: Resolved
    • P2
    • Resolution: Fixed
    • None
    • 2.8.0
    • None

    Description

      Hi,

      beam Filesystem operations copy, rename and delete are void in SDK. Hadoop native filesystem operations are not and returns void. Current implementation in Beam ignores the result and pass as long as exception is not thrown.

      I got burned by this when using 'rename' to do a 'move' operation on HDFS. If target directory does not exists, operations returns false and do not touch the file.

      https://github.com/apache/beam/blob/master/sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystem.java#L148

      Attachments

        Issue Links

          Activity

            People

              timrobertson100 Tim Robertson
              JozoVilcek Jozef Vilcek
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 20m
                  1h 20m