Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25905

BlockManager should expose getRemoteManagedBuffer to avoid creating bytebuffers

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.4.0
    • 3.0.0
    • Spark Core

    Description

      The block manager currently only lets you get a handle on remote data as a ChunkedByteBuffer. But with remote reads of cached data, you really only need an input stream view of the data, which is already available with the ManagedBuffer which is fetched. By forcing conversion to a ChunkedByteBuffer, we end up using more memory than necessary.

      Attachments

        Activity

          People

            wypoon Wing Yew Poon
            irashid Imran Rashid
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: