Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-30602 SPIP: Support push-based shuffle to improve shuffle efficiency
  3. SPARK-35546

Enable push-based shuffle when multiple app attempts are enabled and manage concurrent access to the state in a better way

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.1.0
    • 3.2.0
    • Shuffle
    • None

    Description

      In the current implementation of RemoteBlockPushResolver, two ConcurrentHashmap are used to store #1 applicationId -> mergedShuffleLocalDirPath #2 applicationId+attemptId+shuffleID -> mergedShuffleParitionInfo. As there are four types of messages: ExecutorRegister, PushBlocks, FinalizeShuffleMerge and ApplicationRemove, will trigger different types of operations within these two hashmaps, it is required to maintain strong consistency about the informations stored in these two hashmaps. Otherwise, either there will be data corruption/correctness issues or memory leak in shuffle server. 

      We should come up with systematic way to resolve this, other than spot fixing the potential issues.

      Attachments

        Activity

          People

            zhouyejoe Ye Zhou
            zhouyejoe Ye Zhou
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: