Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-32244

Build and run the Spark with test cases in Github Actions

    XMLWordPrintableJSON

Details

    • Umbrella
    • Status: Resolved
    • Critical
    • Resolution: Done
    • 2.4.6, 3.0.0, 3.1.0
    • 3.1.0
    • Project Infra
    • None

    Description

      Last week and onwards, the Jenkins machines became very unstable for some reasons.

      • Apparently, the machines became extremely slow. Almost all tests can't pass.
      • One machine (worker 4) started to have the corrupt .m2 which fails the build.
      • Documentation build fails time to time for an unknown reason in Jenkins machine specifically.

      Almost all PRs are basically blocked by this instability currently.

      This JIRA aims to run the tests in Github Actions.

      • To avoid depending on few persons who can access to the cluster.
      • To reduce the elapsed time in the build - we could split the tests (e.g., SQL, ML, CORE), and run them in parallel so the total build time will significantly reduce.
      • To control the environment more flexibly.
      • Other contributors can test and propose to fix Github Actions configurations so we can distribute this build management cost.

      Attachments

        Issue Links

          There are no Sub-Tasks for this issue.

          Activity

            People

              gurwls223 Hyukjin Kwon
              gurwls223 Hyukjin Kwon
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: