Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-2503

Lower shuffle output buffer (spark.shuffle.file.buffer.kb)

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • None
    • Shuffle, Spark Core

    Description

      Each shuffle output stream uses a 100KB buffer. This is simply too large. For 10k reducers, 32 cores, that adds up to 32GB in buffer alone.

      The change is simple, but we should make sure there isn't much performance regression when lowering the buffer size.

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-2856 Decrease initial buffer size for Kryo to 64KB

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-2469 Lower shuffle compression buffer memory usage (replace LZF with Snappy for default compression codec)

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #1781 [Github] Pull Request #1781 (rxin)

          Activity

            People

              rxin Reynold Xin
              rxin Reynold Xin
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: