Details
-
Improvement
-
Status: Resolved
-
Minor
-
Resolution: Duplicate
-
3.1.0
-
None
Description
As of now, Spark uses hardcoded value spark_shuffle as the name of the Shuffle Service.
HDP distribution of Spark, on the other hand, uses spark2_shuffle. This is done to be able to run both Spark 1.6 and Spark 2.x on the same Hadoop cluster.
Running vanilla Spark on HDP cluster with only Spark 2.x shuffle service (HDP favor) running becomes impossible due to the shuffle service name mismatch.
Attachments
Issue Links
- is duplicated by
-
SPARK-34828 YARN Shuffle Service: Support configurability of aux service name and service-specific config overrides
- Resolved
- links to