Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Done
-
None
Description
Ensure unique timestamps are generated for commit times with concurrent writers.
this is the piece of code in HoodieActiveTimeline which creates a new commit time.
public static String createNewInstantTime(long milliseconds) { return lastInstantTime.updateAndGet((oldVal) -> { String newCommitTime; do { newCommitTime = HoodieActiveTimeline.COMMIT_FORMATTER.format(new Date(System.currentTimeMillis() + milliseconds)); } while (HoodieTimeline.compareTimestamps(newCommitTime, LESSER_THAN_OR_EQUALS, oldVal)); return newCommitTime; }); }
There are chances that a deltastreamer and a concurrent spark ds writer gets same timestamp and one of them fails.
Related issues and github jiras:
https://github.com/apache/hudi/issues/3782
https://issues.apache.org/jira/browse/HUDI-2549
Attachments
Issue Links
- causes
-
HUDI-2549 Exceptions when using second writer into Hudi table managed by DeltaStreamer
- Closed
- links to