Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-8900

Python SDK should stage only up-to-date versions of pipeline dependencies defined by requirements file.

Details

    • Bug
    • Status: Open
    • P3
    • Resolution: Unresolved
    • None
    • None
    • sdk-py-core

    Description

      When staging pipeline dependencies specified in requirements.txt file and transitive deps, we stage entire content of requirements-cache directory on user's machine, which may include dependencies that were cached a while ago and are no longer needed. See [1] for discussion and some ideas how we can avoid staging packages we don't need.

      [1] https://lists.apache.org/thread.html/f99820489026a46160505ea9a4c8c4a0eabe8649143c1ee22b7088aa%40%3Cdev.beam.apache.org%3E

      Attachments

        Activity

          People

            Unassigned Unassigned
            tvalentyn Valentyn Tymofieiev
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated: