Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-6702

GCP dependencies for the Python SDK are too restrictive

Details

    • Bug
    • Status: Resolved
    • P3
    • Resolution: Fixed
    • 2.12.0
    • Missing
    • sdk-py-core
    • None

    Description

      The fix for BEAM-3324 introduced a bunch of dependencies in the `[gcp]` `extras_require` specification and locked them to specific versions. Unfortunately this has the side-effect of making it probable that you get version conflicts downstream in projects that include `apache_beam` in their `requirements.txt`.

      As it stands I'm running into issues when I ask [`pip-compile`](https://github.com/jazzband/pip-tools) to resolve dependencies with `apache_beam` in the chain (specified like this `-e git+https://github.com/apache/beam.git@1d13199#egg=apache_beam[gcp]&subdirectory=sdks/python`) - example output:

      ```
      Could not find a version that matches google-cloud-core<0.29dev,
      <0.30dev,==0.28.1,>=0.28.0,>=0.29.0
      Tried: 0.20.0, 0.20.0, 0.20.0, 0.20.0, 0.21.0, 0.21.0, 0.21.0, 0.21.0, 0.22.0, 0.22.0, 0.22.0, 0.22.0, 0.22.1, 0.22.1, 0.22.1, 0.22.1, 0.23.0, 0.23.0, 0.23.0, 0.23.0, 0.23.1, 0.23.1, 0.23.1, 0.23.1, 0.24.0, 0.24.0, 0.24.0, 0.24.0, 0.24.1, 0.24.1, 0.24.1, 0.24.1, 0.25.0, 0.25.0, 0.25.0, 0.25.0, 0.26.0, 0.26.0, 0.26.0, 0.26.0, 0.27.0, 0.27.0, 0.27.0, 0.27.0, 0.27.1, 0.27.1, 0.27.1, 0.27.1, 0.28.0, 0.28.0, 0.28.0, 0.28.0, 0.28.1, 0.28.1, 0.28.1, 0.28.1, 0.29.0, 0.29.0, 0.29.0, 0.29.0, 0.29.1, 0.29.1, 0.29.1, 0.29.1
      There are incompatible versions in the resolved dependencies.
      ```
      Best practice is to lock your versions explicitly when you actually use a library (via a projects `requirements.txt`) but leave the requirements specifications as liberal as possible in `setup.py` - this allows some flex in the dependency resolution system so that you have a higher chance of different libraries playing nice.

      An example would be to change a requirement like `'google-cloud-bigtable==0.31.1'` to be `'google-cloud-bigtable>=0.31.1'` (unless you know of an explicit reason why there is an incompatibility with a newer version of the library) in the `setup.py` file.

      Attachments

        Activity

          People

            Unassigned Unassigned
            jkp Jamie Kirkpatrick
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 6h
                6h