RocksDB versionID Mismatch in SST files with Compaction

RocksDB compaction can result in version Id mismatch errors if the same version is committed twice from the same executor. (Multiple commits can happen due to Spark Stage/task retry).

A particular scenario where this can happen is provided below: 

1. Version V1 is loaded on executor A, RocksDB State Store has 195.sst, 196.sst, 197.sst and 198.sst files. 
   2. State changes are made, which result in creation of a new table file 200.sst. 
   3. State store is committed as version V2. The SST file 200.sst (as 000200-8c80161a-bc23-4e3b-b175-cffe38e427c7.sst) is uploaded to DFS, and previous 4 files are reused. A new metadata file is created to track the exact SST files with unique IDs, and uploaded with RocksDB Manifest as part of V1.zip.
   4. Rocks DB compaction is triggered at the same time. The compaction creates a new L1 file (201.sst), and deletes existing 5 SST files.
   5. Spark Stage is retried. 
   6. Version V1 is reloaded on the same executor. The local files are inspected, and 201.sst is deleted. The 4 SST files in version V1 are downloaded again to local file system. 
   7. Any local files which are deleted (as part of version load) are also removed from local → DFS file upload tracking. *However, the files already deleted as a result of compaction are not removed from tracking. This is the bug which resulted in the failure.*
   8. State store is committed as version V1. However, the local mapping of SST files to DFS file path still has 200.sst in its tracking, hence the SST file is not re-uploaded. A new metadata file is created to track the exact SST files with unique IDs, and uploaded with the new RocksDB Manifest as part of V2.zip. (The V2.zip file is overwritten here atomically)
   9. A new executor tried to load version V2. However, the SST files in (1) are now incompatible with Manifest file in (6) resulting in the version Id mismatch failure.

We need to ensure that any files deleted from local filesystem post compaction are not tracked in uploadedDFSFiles mapping if the same version is loaded again.