Details

    Description

      Python 3.12 build fails as below:

      /__w/spark/spark/python/pyspark/pandas/utils.py:1015: PandasAPIOnSparkAdviceWarning: `to_pandas` loads all data into the driver's memory. It should only be used if the resulting pandas Series is expected to be small.
        warnings.warn(message, PandasAPIOnSparkAdviceWarning)
      /__w/spark/spark/python/pyspark/testing/pandasutils.py:401: FutureWarning: `assertPandasOnSparkEqual` will be removed in Spark 4.0.0. Use `ps.testing.assert_frame_equal`, `ps.testing.assert_series_equal` and `ps.testing.assert_index_equal` instead.
        warnings.warn(
      /__w/spark/spark/python/pyspark/pandas/utils.py:1015: PandasAPIOnSparkAdviceWarning: `to_pandas` loads all data into the driver's memory. It should only be used if the resulting pandas DataFrame is expected to be small.
        warnings.warn(message, PandasAPIOnSparkAdviceWarning)
      ok (15.809s)
        test_groupby_rolling_sum (pyspark.pandas.tests.connect.test_parity_ops_on_diff_frames_groupby_rolling.OpsOnDiffFramesGroupByRollingParityTests.test_groupby_rolling_sum) ... ERROR StatusCo
      Had test failures in pyspark.pandas.tests.connect.test_parity_ops_on_diff_frames_groupby_rolling with python3.12; see logs.
      Error:  running /__w/spark/spark/python/run-tests --modules=pyspark-pandas-connect-part2 --parallelism=1 --python-executables=pypy3,python3.10,python3.11,python3.12 ; received return code 255
      Error: Process completed with exit code 19.
      

      https://github.com/apache/spark/actions/runs/7034467411/job/19154767856

      I suspect this by OOM. We could split the build.

      Attachments

        Activity

          People

            gurwls223 Hyukjin Kwon
            gurwls223 Hyukjin Kwon
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: