Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27665

Split fetch shuffle blocks protocol from OpenBlocks

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.4.0
    • 3.0.0
    • Spark Core
    • None

    Description

      As the current approach in OneForOneBlockFetcher, we reuse the OpenBlocks protocol to describe the fetch request for shuffle blocks, and it causes the extension work for shuffle fetching like SPARK-9853 and SPARK-25341 very awkward. We need a new protocol only for shuffle blocks fetcher.

      Attachments

        Issue Links

          is blocked by

          New Feature - A new feature of the product, which has yet to be developed. SPARK-27780 Shuffle server & client should be versioned to enable smoother upgrade

          • Major - Major loss of function.
          • Open
          is related to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-29435 Spark 3 doesnt work with older shuffle service

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #24565

          Activity

            People

              XuanYuan Yuanjian Li
              XuanYuan Yuanjian Li
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: