Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27216

Upgrade RoaringBitmap to 0.7.45 to fix Kryo unsafe ser/dser issue

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.0.0, 2.1.0, 2.2.0, 2.3.3, 2.4.0, 3.0.0
    • 2.3.4, 2.4.2, 3.0.0
    • Spark Core

    Description

      HighlyCompressedMapStatus uses RoaringBitmap to record the empty blocks. But RoaringBitmap-0.5.11 couldn't be ser/deser with unsafe KryoSerializer.

      We can use below UT to reproduce:

        test("kryo serialization with RoaringBitmap") {
    val bitmap = new RoaringBitmap
    bitmap.add(1787)

    val safeSer = new KryoSerializer(conf).newInstance()
    val bitmap2 : RoaringBitmap = safeSer.deserialize(safeSer.serialize(bitmap))
    assert(bitmap2.equals(bitmap))

    conf.set("spark.kryo.unsafe", "true")
    val unsafeSer = new KryoSerializer(conf).newInstance()
    val bitmap3 : RoaringBitmap = unsafeSer.deserialize(unsafeSer.serialize(bitmap))
    assert(bitmap3.equals(bitmap)) // this will fail
  }

      Upgrade to latest version 0.7.45 to fix it

      Attachments

        Issue Links

          is duplicated by

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-27530 FetchFailedException: Received a zero-size buffer for block shuffle

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-23178 Kryo Unsafe problems with count distinct from cache

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-24160 ShuffleBlockFetcherIterator should fail if it receives zero-size blocks

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-27367 Faster RoaringBitmap Serialization with v0.8.0

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-27530 FetchFailedException: Received a zero-size buffer for block shuffle

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #24157

          Web Link GitHub Pull Request #24264

          Web Link GitHub Pull Request #24290

          Web Link GitHub Pull Request #24291

          Activity

            People

              cltlfcjin Lantao Jin
              cltlfcjin Lantao Jin
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: