Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27412

Add a new shuffle manager to use Persistent Memory as shuffle and spilling storage

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Minor
    • Resolution: Won't Fix
    • 3.0.0
    • None
    • Shuffle, Spark Core
    • Patch

    Description

      Add a new shuffle manager called "PmemShuffleManager", by using which, we can use Persistent Memory Device as storage for shuffle and external sorter spilling.

      In this implementation, we leveraged Persistent Memory Development Kit(PMDK) to support transaction write with high performance.

       

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              xuechendi Chendi.Xue
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: