Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-37244

Build and test on Python 3.10

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.3.0
    • 3.3.0
    • PySpark

    Description

      Python 3.10 introduced breaking changes. We need to update PySpark.

      For example, the following.

      >>> from collections import Callable
<stdin>:1: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.10 it will stop working

      This is targeting Apache Spark 3.3.0 in 2022.

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. SPARK-37249 ImportError: cannot import name 'Callable' from 'collections'

          • Major - Major loss of function.
          • Closed
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-37257 Update setup.py for Python 3.10

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-37253 try_simplify_traceback should not fail when tb_frame.f_lineno is None

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #34526 [Github] Pull Request #34526 (dongjoon-hyun)

          Pull request #34529 [Github] Pull Request #34529 (dongjoon-hyun)

          Activity

            People

              dongjoon Dongjoon Hyun
              dongjoon Dongjoon Hyun
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: