Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-45981

Improve Python language test coverage

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      This umbrella Jira aims to improve Apache Spark 4 test coverage across various Python language versions .

      • PySpark
      • Spark Connect Python Client

      Attachments

        Issue Links

        is related to

        Umbrella - An overarching type made of sub-tasks SPARK-44111 Prepare Apache Spark 4.0.0

        • Critical - Crashes, loss of data, severe memory leak.
        • Open
        1.
        		 Add Python GitHub Action Daily Job Sub-task Resolved Dongjoon Hyun Actions
        2.
        		 Add `Python 3.10` to Infra docker image Sub-task Resolved Dongjoon Hyun Actions
        3.
        		 Add `Python 3.10` to the Daily Python Github Action job Sub-task Resolved Dongjoon Hyun Actions
        4.
        		 Add `Python 3.11` to Infra docker image Sub-task Resolved Dongjoon Hyun Actions
        5.
        		 Add `Python 3.11` to the Daily Python Github Action job Sub-task Resolved Dongjoon Hyun Actions
        6.
        		 Fix `pyspark.sql.tests.connect.test_connect_basic` in Python 3.11 Sub-task Resolved Dongjoon Hyun Actions
        7.
        		 Upgrade `protobuf` to 4.25.1 to support `Python 3.11` Sub-task Resolved Dongjoon Hyun Actions
        8.
        		 Upgrade `protobuf-java` to 3.25.1 to match with protobuf 4.25.1 Sub-task Resolved Dongjoon Hyun Actions
        9.
        		 Fix `pyspark.pandas.tests.computation.test_apply_func` in Python 3.11 Sub-task Resolved Dongjoon Hyun Actions
        10.
        		 Fix `pyspark.pandas.tests.connect.computation.test_parity_apply_func` in Python 3.11 Sub-task Resolved Dongjoon Hyun Actions
        11.
        		 Fix `pyspark.ml.torch.tests.test_distributor` in Python 3.11 Sub-task Resolved Hyukjin Kwon Actions
        12.
        		 Add `Python 3.12` to Infra docker image Sub-task Resolved Dongjoon Hyun Actions
        13.
        		 Add `Python 3.12` to the Daily Python Github Action job Sub-task Resolved Hyukjin Kwon Actions
        14.
        		 Upgrade `grpcio*` to 1.59.3 for Python 3.12 Sub-task Resolved Dongjoon Hyun Actions
        15.
        		 Reenable a `releaseSession` test case in SparkConnectServiceE2ESuite Sub-task Resolved Hyukjin Kwon Actions
        16.
        		 Install `six==1.16.0` explicitly for `pandas` in Python 3.12 Sub-task Resolved Dongjoon Hyun Actions
        17.
        		 Remove `unittest` deprecated alias usage for Python 3.12 Sub-task Resolved Dongjoon Hyun Actions
        18.
        		 Install `torch` nightly only at Python 3.12 in Infra docker image Sub-task Resolved Dongjoon Hyun Actions
        19.
        		 Upgrade Cloudpickle to 3.0.0 Sub-task Resolved Hyukjin Kwon Actions
        20.
        		 Upgrade `pytorch` for Python 3.12 Sub-task Resolved Hyukjin Kwon Actions
        21.
        		 Fix the doctest in pyspark.pandas.frame.DataFrame.to_dict (Python 3.12) Sub-task Resolved Hyukjin Kwon Actions
        22.
        		 Flaky `pyspark.tests.test_worker.WorkerSegfaultNonDaemonTest.test_python_segfault` with Python 3.12 Sub-task Resolved Hyukjin Kwon Actions
        23.
        		 Reeanble `pyspark.tests.test_worker.WorkerSegfaultNonDaemonTest.test_python_segfault` with Python 3.12 Sub-task Open Unassigned Actions
        24.
        		 Install torchvision for Python 3.12 build Sub-task Resolved Hyukjin Kwon Actions
        25.
        		 Fix the doctest in pyspark.pandas.series.Series.to_dict (Python 3.12) Sub-task Resolved Hyukjin Kwon Actions
        26.
        		 Fix pyspark.pandas.mlflow.load_model test (Python 3.12) Sub-task Resolved Hyukjin Kwon Actions
        27.
        		 Skip `TorchDistributorLocalUnitTests.test_end_to_end_run_locally` with Python 3.12 Sub-task Resolved Hyukjin Kwon Actions
        28.
        		 Skip 'CrossValidatorTests.test_crossvalidator_with_fold_col' with Python 3.12 Sub-task Resolved Hyukjin Kwon Actions
        29.
        		 Split scheduled Python build Sub-task Resolved Hyukjin Kwon Actions
        30.
        		 Upgrade memory-profiler>=0.61.0 for Python 3.12 Sub-task Resolved Ruifeng Zheng Actions

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            dongjoon Dongjoon Hyun
            dongjoon Dongjoon Hyun
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment