Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-12932

File based sink doesn't respect temp_location pipeline option

Details

    Description

      Unlike FileIO module, theĀ FileBasedSink module doesn't respect the temp_location pipeline option. This isn't expected and documented properly in the respective module's documentation.

      I do understand that the temp_location is technically a pipeline option that's only required by the DataflowRunner but I think it might be worth it to pluck it out of the DataflowRunner and make it be a global pipeline option.

      In addition, I couldn't info on what is the difference between WriteToFiles and WriteToText transforms. It seems that the former uses the FileBasedSink and thus correctly writes temporary files to the temp_location when specified but the latter doesn't do that.

      Attachments

        Activity

          People

            Unassigned Unassigned
            mareksuscak Marek Suscak
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: