Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Duplicate
-
3.1.0
-
None
-
None
Description
RoaringBitmap 0.8.0 adds faster serde, but also requires us to change how we call the serde routines slightly to take advantage of it. This is probably a worthwhile optimization as the every shuffle map task with a large # of partitions generates these bitmaps, and the driver especially has to deserialize many of these messages.
See
Attachments
Issue Links
- is related to
-
SPARK-27216 Upgrade RoaringBitmap to 0.7.45 to fix Kryo unsafe ser/dser issue
- Resolved