Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-22540

HighlyCompressedMapStatus's avgSize is incorrect

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.3.0
    • 2.2.1, 2.3.0
    • Spark Core
    • None

    Description

      The calculation of HighlyCompressedMapStatus's avgSize is incorrect.
      Currently, it looks like "sum of small blocks / count of all non empty blocks", the count of all non empty blocks not only contains small blocks, which contains huge blocks number also, but we need the count of small blocks only.

      Attachments

        Activity

          People

            yucai yucai
            yucai yucai
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: