Memstore flush fencing issue for SFT

The scenarios is the following:

1. rs1 is flushing file to S3 for region1
2. rs1 loses ZK lock
3. region1 gets assigned to rs2
4. rs2 opens region1
5. rs1 completes flush and updates sft file for region1
6. rs2 has a different “version” of the sft file for region1

The flush should fail at the end, but the SFT file gets overwritten before that, resulting in potential data loss.

Potential solutions include:

• Adding timestamp to the tracker file names. This and creating a new tracker file when an rs open the region would allow us to list available tracker files before an update and compare the found timestamps to the one stored in memory to verify the store still owns the latest tracker file
• Using the existing timestamp in the tracker file content. This would also require us to create a new tracker file when a new rs opens the region, but instead of listing the available tracker files, we could try to load and de-serialize the last tracker file and compare the timestamp found in it to the one stored in memory.