Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-44566

Spark CI Improvement

    XMLWordPrintableJSON

Details

    • Umbrella
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 4.0.0
    • None
    • Build, Project Infra, Tests
    • None

    Description

      I have an offline discussion with gurwls223 and LuciferYang, and we think that several points should be improved:

      1. it should be tested with Maven
      2. all supported Python Versions should be tested
      3. clean up unused files ASAP, since the testing resource is quite limited

      To avoid increase the workload too much, we can add daily GA first.

      Attachments

        1.
        		 Daily GA for Maven testing Sub-task Resolved Yang Jie
        2.
        		 Daily GA for Python 3.10 Sub-task Resolved Unassigned
        3.
        		 Daily GA for Python 3.11 Sub-task Resolved Unassigned
        4.
        		 Reuse spark build among pyspark-* modules Sub-task Resolved Unassigned
        5.
        		 Clean up unused installers ASAP Sub-task Resolved Ruifeng Zheng
        6.
        		 TorchDistributor should install cpu-only Torch for testing Sub-task Resolved Ruifeng Zheng
        7.
        		 Make `breaking-changes-buf` cancelable Sub-task Resolved Ruifeng Zheng
        8.
        		 Make `repl` module daily test pass Sub-task Resolved Yang Jie
        9.
        		 Make `connect` module daily test pass Sub-task Resolved Yang Jie
        10.
        		 Make `hive-thriftserver` module daily test pass Sub-task Resolved Yang Jie
        11.
        		 Free up disk space for non-container jobs Sub-task Resolved Ruifeng Zheng
        12.
        		 Free up disk space for container jobs Sub-task Resolved Ruifeng Zheng
        13.
        		 Uninstall CodeQL/Go/Node in non-container jobs Sub-task Resolved Ruifeng Zheng
        14.
        		 Uninstall large ML libraries for non-ML jobs Sub-task Resolved Ruifeng Zheng
        15.
        		 Optimize apt-get install in Dockerfile Sub-task Resolved Ruifeng Zheng
        16.
        		 Make utils.eventually a parameterized decorator Sub-task Resolved Ruifeng Zheng
        17.
        		 re-org apt-get installations Sub-task Resolved Ruifeng Zheng
        18.
        		 re-org R package installations Sub-task Resolved Ruifeng Zheng
        19.
        		 Cache python deps for linter and documentation Sub-task Open Unassigned
        20.
        		 Re-org the testing dockerfile Sub-task Resolved Ruifeng Zheng
        21.
        		 merge pyspark-error to pyspark-core Sub-task Resolved Ruifeng Zheng
        22.
        		 Cache the Python dependencies for SQL tests Sub-task Open Unassigned
        23.
        		 Purge pip cache in dockerfile Sub-task Resolved Ruifeng Zheng
        24.
        		 Reduce the number of layers of testing dockerfile Sub-task Resolved Ruifeng Zheng

        Activity

          People

            Unassigned Unassigned
            podongfeng Ruifeng Zheng
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: