Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-14493

HdfsDownloader gets wrong range

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • P2
    • Resolution: Fixed
    • 2.31.0
    • Not applicable
    • io-py-hadoop
    • None

    Description

      Trying to read avro data from HDFS from a python sidecar worker fails with:

      File "python3.7/site-packages/apache_beam/io/filesystemio.py", line 123, in readinto
      b[:len(data)] = data
      ValueError: memoryview assignment: lvalue and rvalue have different structures

      This is the same issue as https://issues.apache.org/jira/browse/BEAM-9152 which was marked as resolved without being resolved at all.

       As remarked by Jean-Christophe CARLES on https://issues.apache.org/jira/browse/BEAM-9152 , patching hadoopfilesystem.py by removing " + 1" in HdfsDownloader.get_range fixes it for us.

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. BEAM-9152 Hadoop Downloader Range not correct

          • P1 - Critical bug: data loss, total loss of function, or loss of testing signal due to test failures or flakiness
          • Triage Needed

          Activity

            People

              Unassigned Unassigned
              Enzo90910 Vincent Bernardi
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: