Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-1251 Python 3 Support
  3. BEAM-7474

Add SDK harness containers for Py 3.6, Py 3.7

Details

    • Sub-task
    • Status: Resolved
    • P2
    • Resolution: Fixed
    • None
    • 2.16.0
    • sdk-py-harness
    • None

    Description

      Currently we can build a Py3-compatible container image with gradle by running:

      ./gradlew :sdks:python:container:py3:docker

      This builds a docker container image like: valentyn-docker-apache.bintray.io/beam/python3

      The code for this is defined in: https://github.com/apache/beam/blob/ae60a72b03f3a2b6b2a06667ec1868a7acc8e38f/sdks/python/container/py3/build.gradle#L48

      To support portable runners that use a container (e.g. Flink) on multiple versions of Python 3, we should make it possible to build Python 3-compatible SDK harness containers bundled with any desired python version.

      We could have several gradle projects:

      :sdks:python:container:py35:docker
      :sdks:python:container:py36:docker
      :sdks:python:container:py37:docker

      and several Dockerfiles to support this:

      sdks/python/container/py35/Dockerfile
      sdks/python/container/py36/Dockerfile
      sdks/python/container/py37/Dockerfile

      The only difference right now would be the base image used in FROM field in Dockerfile.

      Alternatively, we could have one parameterized Dockerfile that starts with :

      ARG BASE_IMAGE
      FROM $BASE_IMAGE
      ...
      

      I think the latter approach, may result in complications later if these containers will need to diverge down the road.

      cc'ing a few folks who may have some feedback on this: angoenka mxm robertwb Juta frederik.

      Attachments

        Issue Links

          Activity

            People

              hannahjiang Hannah Jiang
              tvalentyn Valentyn Tymofieiev
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: