Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-1251

Python 3 Support

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Triage Needed
    • P2
    • Resolution: Fixed
    • None
    • 2.11.0
    • sdk-py-core
    • None

    Description

      FAQ

      Does Apache Beam support Python 3?

      • Yes!

      Is there any remaining work?

      Which SDK version should I use?

      • For best experience, use the latest released SDK. For summary of Py3-related changes, read this thread.

      Help! I am getting a pickling error in StockUnpickler.find_class() on Python 3.

      • Does the error happens in load_session call? See BEAM-6158 .
      • Do you use Beam SDK less than 2.17.0? See BEAM-8651.

      My Avro Sink no longer works.

      • Beam switched to use FastAvro as a default library on Python 3. The fastavro-based Avro sink expects schema as a dictionary, while the avro-python3-based Avro Sink expects a schema that was previously parsed by avro.schema.Parse(). Fastavro will not accept a schema parsed by avro-python3, so make sure you pass the correct schema. See: BEAM-10769.

      My streaming pipelines are stuck on Python 3.

      • Do you use Beam SDK less than 2.17.0? If so please upgrade to 2.17.0. See BEAM-8651.

      Attachments

        Issue Links

          duplicates

          Improvement - An improvement or enhancement to an existing feature or task. BEAM-1373 Update Python SDK code to support both Python 2 and 3

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved
          incorporates

          Sub-task - The sub-task of the issue BEAM-4000 Futurize and fix python 2 compatibility for io subpackage

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Sub-task - The sub-task of the issue BEAM-4002 Futurize and fix python 2 compatibility for options subpackage

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Sub-task - The sub-task of the issue BEAM-4004 Futurize and fix python 2 compatibility for testing subpackage

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Sub-task - The sub-task of the issue BEAM-4005 Futurize and fix python 2 compatibility for tools subpackage

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Sub-task - The sub-task of the issue BEAM-4006 Futurize and fix python 2 compatibility for transforms subpackage

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Sub-task - The sub-task of the issue BEAM-4009 Futurize and fix python 2 compatibility for unpackaged files

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Sub-task - The sub-task of the issue BEAM-4511 Create a tox environment that uses Py3 interpreter for pre/post commit test suites, once codebase supports Py3.

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. BEAM-3381 Handle trivial inference with default arguments

          • P3 - Non-urgent bugs, features, and improvements
          • Open

          Bug - A problem which impairs or prevents the functions of the product. BEAM-3730 typehints.TypeVariable issues with __hash__

          • P3 - Non-urgent bugs, features, and improvements
          • Open

          Bug - A problem which impairs or prevents the functions of the product. BEAM-4229 PostCommits broken on SpannerReadIT tests

          • P0 - Outage blocking development and/or testing work; requires immediate and continuous attention
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. BEAM-1368 Python setup should be portable to Python 3.x

          • P4 - Trivial items, spelling errors, etc.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. BEAM-3671 Need Python 3.x interpreter in Jenkins machines

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. BEAM-1373 Update Python SDK code to support both Python 2 and 3

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. BEAM-2784 Fix issues from automated conversion to allow Python 2 functionality

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. BEAM-2786 Update jenkins test scripts to test with Py2 & Py3

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. BEAM-3233 Fix Py3 list comprehension type inferance

          • P3 - Non-urgent bugs, features, and improvements
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. BEAM-3761 Fix Python 3 cmp function

          • P3 - Non-urgent bugs, features, and improvements
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. BEAM-4715 Have Beam tests running and passing on Python 3

          • P3 - Non-urgent bugs, features, and improvements
          • Resolved

          Task - A task that needs to be done. BEAM-3380 Make type inference compatible with Python 3.6

          • P3 - Non-urgent bugs, features, and improvements
          • Open

          Task - A task that needs to be done. BEAM-3781 Figure out min supported Python 3 version

          • P3 - Non-urgent bugs, features, and improvements
          • Open
          is a parent of

          Sub-task - The sub-task of the issue BEAM-4511 Create a tox environment that uses Py3 interpreter for pre/post commit test suites, once codebase supports Py3.

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Sub-task - The sub-task of the issue BEAM-6877 TypeHints Py3 Error: Type inference tests fail on Python 3.6 due to bytecode changes

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Sub-task - The sub-task of the issue BEAM-5720 Default coder breaks with large ints on Python 3

          • P3 - Non-urgent bugs, features, and improvements
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. BEAM-5255 Fix over-aggressive division futurization.

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. BEAM-6983 Python 3.6 Support

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. BEAM-6984 Python 3.7 Support

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. BEAM-4543 Remove dependency on googledatastore in favor of google-cloud-datastore.

          • P3 - Non-urgent bugs, features, and improvements
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. BEAM-8494 Python 3.8 Support

          • P3 - Non-urgent bugs, features, and improvements
          • Triage Needed
          is blocked by

          Improvement - An improvement or enhancement to an existing feature or task. BEAM-2784 Fix issues from automated conversion to allow Python 2 functionality

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. BEAM-8441 Python 3 pipeline fails with errors in StockUnpickler.find_class() during loading a main session.

          • P3 - Non-urgent bugs, features, and improvements
          • Open
          links to

          Web Link GitHub Pull Request #4570

          Web Link GitHub Pull Request #4571

          Web Link GitHub Pull Request #4697

          Web Link GitHub Pull Request #4804

          Web Link GitHub Pull Request #4820

          Web Link GitHub Pull Request #4834

          Web Link GitHub Pull Request #4835

          Web Link GitHub Pull Request #5531

          Web Link GitHub Pull Request #5842

          Web Link GitHub Pull Request #5869

          Web Link GitHub Pull Request #5887

          Web Link GitHub Pull Request #5890

          Web Link GitHub Pull Request #5908

          Web Link GitHub Pull Request #5911

          Web Link GitHub Pull Request #6166

          Web Link GitHub Pull Request #6489

          Web Link GitHub Pull Request #6650

          Web Link GitHub Pull Request #6678

          Web Link GitHub Pull Request #6679

          Web Link GitHub Pull Request #7446

          Web Link GitHub Pull Request #7953

          Web Link GitHub Pull Request #8956

          Web Link GitHub Pull Request #9537

          Web Link GitHub Pull Request #14198

          Activity

            People

              tvalentyn Valentyn Tymofieiev
              eyad.alsibai@gmail.com Eyad Sibai
              Votes:
              39 Vote for this issue
              Watchers:
              46 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 516h 40m
                  516h 40m