Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-45981 Improve Python language test coverage
  3. SPARK-46059

Install `six==1.16.0` explicitly for `pandas` in Python 3.12

    XMLWordPrintableJSON

Details

    Description

      This happens in Python 3.12 CI only.

      • https://github.com/apache/spark/actions/runs/6959106836/job/18935673389
        Starting test(python3.12): pyspark.streaming.tests.test_context (temp output: /__w/spark/spark/python/target/73ed28d0-ae18-426e-9760-d03bea982a9b/python3.12__pyspark.streaming.tests.test_context__l4z6a7a2.log)
        Traceback (most recent call last):
          File "<frozen runpy>", line 198, in _run_module_as_main
          File "<frozen runpy>", line 88, in _run_code
          File "/__w/spark/spark/python/pyspark/streaming/tests/test_context.py", line 23, in <module>
            from pyspark.testing.streamingutils import PySparkStreamingTestCase
          File "/__w/spark/spark/python/pyspark/testing/__init__.py", line 19, in <module>
            from pyspark.testing.pandasutils import assertPandasOnSparkEqual
          File "/__w/spark/spark/python/pyspark/testing/pandasutils.py", line 58, in <module>
            import pyspark.pandas as ps
          File "/__w/spark/spark/python/pyspark/pandas/__init__.py", line 33, in <module>
            require_minimum_pandas_version()
          File "/__w/spark/spark/python/pyspark/sql/pandas/utils.py", line 27, in require_minimum_pandas_version
            import pandas
          File "/usr/local/lib/python3.12/dist-packages/pandas/__init__.py", line 46, in <module>
            from pandas.core.api import (
          File "/usr/local/lib/python3.12/dist-packages/pandas/core/api.py", line 1, in <module>
            from pandas._libs import (
          File "/usr/local/lib/python3.12/dist-packages/pandas/_libs/__init__.py", line 18, in <module>
            from pandas._libs.interval import Interval
          File "interval.pyx", line 1, in init pandas._libs.interval
          File "hashtable.pyx", line 1, in init pandas._libs.hashtable
          File "missing.pyx", line 42, in init pandas._libs.missing
        AttributeError: partially initialized module 'pandas' has no attribute '_pandas_datetime_CAPI' (most likely due to a circular import)
        

      Attachments

        Issue Links

          Activity

            People

              dongjoon Dongjoon Hyun
              dongjoon Dongjoon Hyun
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: