Guarantee compatibility of snapshot (live entities, KVstore entities)

This issue is a follow-up issue after SPARK-29111 and SPARK-29261, which both issues WILL NOT guarantee compatibility.

To safely clean up old event log files after snapshot has been written for these files, we have to ensure the snapshot file can restore the state as same as we replay from these event log files. The issue is on migrating to the newer Spark version - if snapshot is not readable due to incompatibility, the app cannot be read entirely as we've already removed old event log files. If we can guarantee compatibility we can move on to the next item, cleaning up old event log files to save space.