Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-47702

Shuffle service endpoint is not removed from the locations list when RDD block is removed form a node.

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.5.1
    • None
    • Spark Core

    Description

      If SHUFFLE_SERVICE_FETCH_RDD_ENABLED is set to true, driver stores both executor end point and the external shuffle end points for a RDD block. When the RDD is migrated, the location info is updated to add the end point corresponds to new location and the old end point is removed. But currently, only the executor end point is removed. The shuffle service end point is not removed. This cause failure during RDD read if the shuffle service end point is chosen due to task locality.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              maheshk114 mahesh kumar behera
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated: