Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27651

Avoid the network when block manager fetches shuffle blocks from the same host

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0
    • 3.0.0
    • Block Manager, Spark Core
    • None

    Description

      When a shuffle block (content) is fetched the network is always used even when it is fetched from the external shuffle service running on the same host. This can be avoided by getting the local directories of the same host executors from the external shuffle service and accessing those blocks from the disk directly.

      Attachments

        Issue Links

          causes

          Bug - A problem which impairs or prevents the functions of the product. SPARK-30604 HostLocal Block size missed in log total bytes

          • Trivial - Cosmetic problem like misspelt words or misaligned text.
          • Resolved
          depends upon

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-27622 Avoid the network when block manager fetches disk persisted RDD blocks from the same host

          • Major - Major loss of function.
          • Resolved
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-31055 Update config docs for shuffle local host reads to have dep on external shuffle service

          • Major - Major loss of function.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-6521 Bypass network shuffle read if both endpoints are local

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-32077 Support host-local shuffle data reading with external shuffle service disabled

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #25299

          Activity

            People

              attilapiros Attila Zsolt Piros
              attilapiros Attila Zsolt Piros
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: