Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-29106

Add jenkins arm test for spark

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Test
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • 3.0.0
    • None
    • Tests
    • None

    Description

      Add arm test jobs to amplab jenkins for spark.

      Till now we made two arm test periodic jobs for spark in OpenLab, one is based on master with hadoop 2.7(similar with QA test of amplab jenkins), other one is based on a new branch which we made on date 09-09, see  http://status.openlabtesting.org/builds/job/spark-master-unit-test-hadoop-2.7-arm64  and http://status.openlabtesting.org/builds/job/spark-unchanged-branch-unit-test-hadoop-2.7-arm64. We only have to care about the first one when integrate arm test with amplab jenkins.

      About the k8s test on arm, we have took test it, see https://github.com/theopenlab/spark/pull/17, maybe we can integrate it later. 

      And we plan test on other stable branches too, and we can integrate them to amplab when they are ready.

      We have offered an arm instance and sent the infos to shane knapp, thanks shane to add the first arm job to amplab jenkins  

      The other important thing is about the leveldbjni https://github.com/fusesource/leveldbjni, spark depends on leveldbjni-all-1.8 https://mvnrepository.com/artifact/org.fusesource.leveldbjni/leveldbjni-all/1.8, we can see there is no arm64 supporting. So we build an arm64 supporting release of leveldbjni see https://mvnrepository.com/artifact/org.openlabtesting.leveldbjni/leveldbjni-all/1.8, but we can't modified the spark pom.xml directly with something like 'property'/'profile' to choose correct jar package on arm or x86 platform, because spark depends on some hadoop packages like hadoop-hdfs, the packages depend on leveldbjni-all-1.8 too, unless hadoop release with new arm supporting leveldbjni jar. Now we download the leveldbjni-al-1.8 of openlabtesting and 'mvn install' to use it when arm testing for spark.

      PS: The issues found and fixed:
      SPARK-28770
      https://github.com/apache/spark/pull/25673
       
      SPARK-28519
      https://github.com/apache/spark/pull/25279
       
      SPARK-28433
      https://github.com/apache/spark/pull/25186

       

      SPARK-28467

      https://github.com/apache/spark/pull/25864

       

      SPARK-29286

      https://github.com/apache/spark/pull/26021

       

       

      Attachments

        1. SparkR-and-pyspark36-testing.txt
          3 kB
          zhao bo
        2. arm-python36.txt
          0.1 kB
          Shane Knapp
        3. R-libs.txt
          2 kB
          Shane Knapp
        4. R-ansible.yml
          2 kB
          Shane Knapp

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            shaneknapp Shane Knapp
            huangtianhua huangtianhua

            Dates

              Created:
              Updated:

              Slack

                Issue deployment