Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-29330

Allow users to chose the name of Spark Shuffle service

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Duplicate
    • 3.1.0
    • 3.2.0
    • Spark Core, YARN
    • None

    Description

      As of now, Spark uses hardcoded value spark_shuffle as the name of the Shuffle Service.

      HDP distribution of Spark, on the other hand, uses spark2_shuffle. This is done to be able to run both Spark 1.6 and Spark 2.x on the same Hadoop cluster.

      Running vanilla Spark on HDP cluster with only Spark 2.x shuffle service (HDP favor) running becomes impossible due to the shuffle service name mismatch.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              nonsleepr Alexander Bessonov
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: