Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-17116

RBF: Update invoke millisecond time as monotonicNow() in RouterSafemodeService.

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.4.0
    • 3.4.0
    • rbf
    • Reviewed

    Description

      The following exceptions occurred in our online environment:

      1. After the machine restarts, the system time is abnormal, is a time in the future
      2. After starting the router, there is log "safemode exit for 24981702 milliseconds...", which has been in the safemode state,
        this is mainly because the startupTime is recorded as the future system time when router is started at this time, and the system time returns to normal soon, resulting in a negative delta,
        at this time, the service can only be restored by restart the router service.

      The relevant logs are:

      2023-07-15 03:15:49,276 INFO  ipc.Server xxx
      2023-07-15 11:21:03,785 INFO  router.DFSRouter (LogAdapter.java:info(51)) [main] - STARTUP_MSG:
      /************************************************************
      STARTUP_MSG: Starting Router
      ...
      2023-07-15 11:21:51,325 INFO xxx
      2023-07-15 03:22:00,257 INFO xxx
      2023-07-15 03:22:29,829 INFO router.RouterSafemodeService (RouterSafemodeService.java:periodicInvoke(167)) [RouterSafemodeService-0] - Delaying safemode exit for 28761777 milliseconds...
      

      Maybe we can be compatible with this case at the code level, and reset the startupTime and enterSafeModeTime in the case of a negative delta,
      which can ensure that the router service can also exit the safemode state normally after the system time returns to normal.

      Attachments

        Issue Links

          Activity

            People

              haiyang Hu Haiyang Hu
              haiyang Hu Haiyang Hu
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: