Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-5036

Optimize FileBasedSink's WriteOperation.moveToOutput()

Details

    • Improvement
    • Status: Resolved
    • P2
    • Resolution: Fixed
    • 2.5.0
    • 2.9.0
    • io-java-files
    • None

    Description

      moveToOutput() methods in FileBasedSink.WriteOperation implements move by copy+delete. It would be better to use a rename() which can be much more effective for some filesystems.

      Filesystem must support cross-directory rename. BEAM-4861 is related to this for the case of HDFS filesystem.

      Feature was discussed here:

      http://mail-archives.apache.org/mod_mbox/beam-dev/201807.mbox/%3CCAF9t7_4Mp54pQ+vRrJrBh9Vx0=uaKnuPZD_qdh_QDm9VXLLsZw@mail.gmail.com%3E

      Attachments

        Activity

          People

            timrobertson100 Tim Robertson
            JozoVilcek Jozef Vilcek
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 14h 20m
                14h 20m