Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Won't Do
-
3.2.0
-
None
-
None
Description
Sometimes when run a SQL job with push based shuffle, exception occurs as below. It seems that there’s no element in the bitmaps which stores merge chunk meta.
Is it a bug that we should not createChunkBlockInfos when bitmaps is empty or the bitmaps should never be empty here ?
Caused by: java.lang.ArithmeticException: / by zero at org.apache.spark.storage.PushBasedFetchHelper.createChunkBlockInfosFromMetaResponse(PushBasedFetchHelper.scala:117) at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:980) at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:84)
related code:
def createChunkBlockInfosFromMetaResponse(
shuffleId: Int,
shuffleMergeId: Int,
reduceId: Int,
blockSize: Long,
bitmaps: Array[RoaringBitmap]): ArrayBuffer[(BlockId, Long, Int)] = {
val approxChunkSize = blockSize / bitmaps.length
val blocksToFetch = new ArrayBuffer[(BlockId, Long, Int)]()
for (i <- bitmaps.indices) {
val blockChunkId = ShuffleBlockChunkId(shuffleId, shuffleMergeId, reduceId, i)
chunksMetaMap.put(blockChunkId, bitmaps(i))
logDebug(s"adding block chunk $blockChunkId of size $approxChunkSize")
blocksToFetch += ((blockChunkId, approxChunkSize, SHUFFLE_PUSH_MAP_ID))
}
blocksToFetch
}
Attachments
Issue Links
- is fixed by
-
SPARK-37675 Prevent overwriting of push shuffle merged files once the shuffle is finalized
-
- Resolved
-