Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-7854

Reading files from local file system does not fully support glob

Details

    • Bug
    • Status: Resolved
    • P2
    • Resolution: Fixed
    • None
    • 2.16.0
    • sdk-java-core

    Description

      Folder structure:   

      A
          B
              a=100
                  data1
                      file1.zst
                      file2.zst 
              a=999 
                  data2
                      file6.zst
              a=397
                  data3
                      file7.zst

       

      Glob:

       

      /A/B/a=[0-9][0-9][0-9]/*/*

      Code:  

       

      input.apply(Create.of(patterns))
           .apply("Matching patterns", FileIO.matchAll())
           .apply(FileIO.readMatches());
      

       

      input is of type PBegin.

      The above code matches 0 files even though, from the glob, its clear it should match all files. I suspect its because of line 227, where only the first parent folder is checked while is could be an asterix in a glob. I believe the right behaviour should be to check all parent folder and use the first one that exists.

      Attachments

        Issue Links

          Activity

            People

              tomerr90 Tomer Zeltzer
              tomerr90 Tomer Zeltzer
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 4h 50m
                  4h 50m