Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-38396

Improve K8s Integration Tests

    XMLWordPrintableJSON

Details

    • Umbrella
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.3.0
    • 3.3.0
    • Kubernetes, Tests

    Description

      This JIRA aims to improve K8s integration tests for the following recent activity.

      • Support Java 17 / Native Apple Silicon
      • Cloud K8s testing (including Graviton instances and K8s 1.22 GA on EKS on March 15th)
      • New K8s model (K8s clients)
      • New custom schedulers and statefulset

      Attachments

        Issue Links

          relates to

          New Feature - A new feature of the product, which has yet to be developed. SPARK-33772 Build and Run Spark on Java 17

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. SPARK-35781 Support Spark on Apple Silicon on macOS natively on Java 17

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-36057 SPIP: Support Customized Kubernetes Schedulers

          • Major - Major loss of function.
          • Resolved
          1.
          		 Support docker-desktop K8S integration test in SBT Sub-task Resolved William Hyun
          2.
          		 Add IntegrationTestBackend.describePods to support all K8s test backends Sub-task Resolved William Hyun
          3.
          		 Support K8s namespace parameter in SBT K8s IT Sub-task Resolved William Hyun
          4.
          		 Support K8s imageTag parameter in SBT K8s IT Sub-task Resolved William Hyun
          5.
          		 Add SBT commands to K8s IT readme Sub-task Resolved William Hyun
          6.
          		 Support K8s image building with Java 17 Sub-task Resolved Dongjoon Hyun
          7.
          		 Use relativePath for K8s remote file test in BasicTestsSuite Sub-task Resolved Dongjoon Hyun
          8.
          		 Support cloud-backend in K8s IT with SBT Sub-task Resolved Dongjoon Hyun
          9.
          		 Add `priorityClassName` integration test case Sub-task Resolved Dongjoon Hyun
          10.
          		 Upgrade the minimum Minikube version to 1.18.0 Sub-task Resolved Dongjoon Hyun
          11.
          		 Fix driver environment verification in BasicDriverFeatureStepSuite Sub-task Resolved Dongjoon Hyun
          12.
          		 Make the Java version installed on the container image used by the K8s integration tests with SBT configurable Sub-task Resolved Kousuke Saruta
          13.
          		 Support K8s integration tests for Java 17 Sub-task Resolved Kousuke Saruta
          14.
          		 Add `spark-` prefix to namespaces and `-driver` suffix to drivers during IT Sub-task Resolved Martin Tzvetanov Grigorov
          15.
          		 Close KubernetesClient in K8S integrations tests Sub-task Resolved Martin Tzvetanov Grigorov
          16.
          		 Upgrade kubernetes-client to 5.12.1 Sub-task Resolved Yikun Jiang
          17.
          		 Use docker-desktop instead of docker-for-desktop for Docker K8S IT deployMode and context name Sub-task Resolved Yikun Jiang
          18.
          		 Use `rbac.authorization.k8s.io/v1` instead of `v1beta1` Sub-task Resolved Yikun Jiang
          19.
          		 Support pyDockerfile and rDockerfile in SBT K8s IT Sub-task Resolved Yikun Jiang
          20.
          		 Word spell error - "labeled" spells as "labled" Sub-task Resolved Qian Sun
          21.
          		 Use Java 17 in K8S integration tests when setting spark-tgz Sub-task Resolved Qian Sun

          Activity

            People

              william William Hyun
              dongjoon Dongjoon Hyun
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: