Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-10985

Avoid passing evicted blocks throughout BlockManager / CacheManager

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • None
    • 2.0.0
    • Block Manager, Spark Core
    • None

    Description

      This is a minor refactoring task.

      Currently when we attempt to put a block in, we get back an array buffer of blocks that are dropped in the process. We do this to propagate these blocks back to our TaskContext, which will add them to its TaskMetrics so we can see them in the SparkUI storage tab properly.

      Now that we have TaskContext.get, we can just use that to propagate this information. This simplifies a lot of the signatures and gets rid of weird return types like the following everywhere:

      ArrayBuffer[(BlockId, BlockStatus)]
      

      Attachments

        Activity

          People

            joshrosen Josh Rosen
            andrewor14 Andrew Or
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: