Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
2.4.0
-
None
Description
The amount of memory used by the broadcast variable is not synchronized to the UI display，
spark-sql> select /+ broadcast(a)/ a.id,b.id from a join b on a.id = b.id;
View the app's driver log：
2019-01-25 16:45:23,726 INFO org.apache.spark.storage.BlockManagerInfo: Added broadcast_4_piece0 in memory on 10.43.xx.xx:33907 (size: 6.6 KB, free: 2.5 GB)
2019-01-25 16:45:23,727 INFO org.apache.spark.storage.BlockManagerInfo: Added broadcast_4_piece0 in memory on 10.43.xx.xx:38399 (size: 6.6 KB, free: 2.5 GB)
2019-01-25 16:45:23,745 INFO org.apache.spark.storage.BlockManagerInfo: Added broadcast_3_piece0 in memory on 10.43.xx.xx:33907 (size: 32.1 KB, free: 2.5 GB)
2019-01-25 16:45:23,749 INFO org.apache.spark.storage.BlockManagerInfo: Added broadcast_3_piece0 in memory on 10.43.xx.xx:38399 (size: 32.1 KB, free: 2.5 GB)
2019-01-25 16:45:23,838 INFO org.apache.spark.storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on 10.43.xx.xx:38399 (size: 147.0 B, free: 2.5 GB)
2019-01-25 16:45:23,840 INFO org.apache.spark.storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on 10.43.xx.xx:33907 (size: 147.0 B, free: 2.5 GB)
Web UI does not have the use of memory，
| Executor ID | Address | Status | RDD Blocks | Storage Memory | Disk Used | Cores | Active Tasks | Failed Tasks | Complete Tasks | Total Tasks | Task Time (GC Time) | Input | Shuffle Read | Shuffle Write | Logs | Thread Dump |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | xxx:38399 | Active | 0 | 0.0 B / 2.7 GB | 0.0 B | 1 | 0 | 0 | 2 | 2 | 4 s (0.4 s) | 8 B | 0.0 B | 0.0 B | ||
| driver | xxx:47936 | Active | 0 | 0.0 B / 384.1 MB | 0.0 B | 0 | 0 | 0 | 0 | 0 | 0.0 ms (0.0 ms) | 0.0 B | 0.0 B | 0.0 B | ||
| 1 | xxx:47414 | Active | 0 | 0.0 B / 2.7 GB | 0.0 B | 1 | 0 | 0 | 0 | 0 | 0.0 ms (0.0 ms) | 0.0 B | 0.0 B | 0.0 B | ||
| 2 | xxx:33907 | Active | 0 | 0.0 B / 2.7 GB | 0.0 B | 1 | 0 | 0 | 2 | 2 | 4 s (0.2 s) | 4 B | 0.0 B | 0.0 B |
Attachments
Issue Links
- links to