Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-1251 Python 3 Support
  3. BEAM-7474

Add SDK harness containers for Py 3.6, Py 3.7

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.16.0
    • Component/s: sdk-py-harness
    • Labels:
      None

      Description

      Currently we can build a Py3-compatible container image with gradle by running:

      ./gradlew :sdks:python:container:py3:docker

      This builds a docker container image like: valentyn-docker-apache.bintray.io/beam/python3

      The code for this is defined in: https://github.com/apache/beam/blob/ae60a72b03f3a2b6b2a06667ec1868a7acc8e38f/sdks/python/container/py3/build.gradle#L48

      To support portable runners that use a container (e.g. Flink) on multiple versions of Python 3, we should make it possible to build Python 3-compatible SDK harness containers bundled with any desired python version.

      We could have several gradle projects:

      :sdks:python:container:py35:docker
      :sdks:python:container:py36:docker
      :sdks:python:container:py37:docker

      and several Dockerfiles to support this:

      sdks/python/container/py35/Dockerfile
      sdks/python/container/py36/Dockerfile
      sdks/python/container/py37/Dockerfile

      The only difference right now would be the base image used in FROM field in Dockerfile.

      Alternatively, we could have one parameterized Dockerfile that starts with :

      ARG BASE_IMAGE
      FROM $BASE_IMAGE
      ...
      

      I think the latter approach, may result in complications later if these containers will need to diverge down the road.

      cc'ing a few folks who may have some feedback on this: Ankur Goenka Maximilian Michels Robert Bradshaw Juta Staes Frederik Bode.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                hannahjiang Hannah Jiang
                Reporter:
                tvalentyn Valentyn Tymofieiev
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: