Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-2559

Ensure unique timestamps are generated for commit times with concurrent writers

    XMLWordPrintableJSON

Details

    Description

      Ensure unique timestamps are generated for commit times with concurrent writers.

      this is the piece of code in HoodieActiveTimeline which creates a new commit time.

      public static String createNewInstantTime(long milliseconds) {
        return lastInstantTime.updateAndGet((oldVal) -> {
          String newCommitTime;
          do {
            newCommitTime = HoodieActiveTimeline.COMMIT_FORMATTER.format(new Date(System.currentTimeMillis() + milliseconds));
          } while (HoodieTimeline.compareTimestamps(newCommitTime, LESSER_THAN_OR_EQUALS, oldVal));
          return newCommitTime;
        });
      }
      

      There are chances that a deltastreamer and a concurrent spark ds writer gets same timestamp and one of them fails. 

      Related issues and github jiras: 

      https://github.com/apache/hudi/issues/3782

      https://issues.apache.org/jira/browse/HUDI-2549

       

       

       

      Attachments

        Issue Links

          Activity

            People

              shivnarayan sivabalan narayanan
              shivnarayan sivabalan narayanan
              Shiyan Xu
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: