Affects Version/s: 2.4.3
Fix Version/s: None
Component/s: Structured Streaming
spark standalone cluster 2.4.3
the folder in Driver
takes up all the space in /tmp after runing spark structured stream job a long time.
it is mainly under the offsets and commits folders.but when I watch it by us command
it got more than 600M,but when We use command
it got 400K.
I think it is because when the file is deleted,it is still used in job.
It wasn't released only if the job is stopped.
How can I solve it?
Is there something wrong here?