Problem

BatchingStateChangeUploadScheduler will upload many empty changelog files (file size == 1  and only contains compressed flag).

These files are not referenced by any checkpoints, are not cleaned up, and become more numerous as the job runs. Taking our big job as an example, 2292 such files were generated within 7 hours. It only takes about 4 months and the number of files in the changelog directory will exceed a million.

Problem causes

This problem is caused by BatchingStateChangeUploadScheduler#drainAndSave not checking whether the task collection is empty. The data in the scheduled queue may have been uploaded when the BatchingStateChangeUploadScheduler#drainAndSave method is executed.

So we should check whether the task collection is empty in BatchingStateChangeUploadScheduler#drainAndSave . WDYT Roman Khachatryan , Yanfei Lei ?