Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-21806

add an option to roll WAL on very slow syncs

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.0.0-alpha-1
    • None
    • None
    • Hide
      HBase RS will now roll the WAL file upon a slow sync call. hbase.regionserver.hlog.roll.on.sync.ms config has been added to HBase, with the default value of 10000 (10 seconds); if a sync of the WAL takes longer than that the file will be replaced under the assumption that we will get a faster datanode pipeline for a new file. This setting works with an existing timeout setting that will still kick in on very long syncs (5 minutes by default) and fail the operation.
      Show
      HBase RS will now roll the WAL file upon a slow sync call. hbase.regionserver.hlog.roll.on.sync.ms config has been added to HBase, with the default value of 10000 (10 seconds); if a sync of the WAL takes longer than that the file will be replaced under the assumption that we will get a faster datanode pipeline for a new file. This setting works with an existing timeout setting that will still kick in on very long syncs (5 minutes by default) and fail the operation.

    Description

      In large heterogeneous clusters sometimes a slow datanode can cause WAL syncs to be very slow. In this case, before the bad datanode recovers, or is discovered and repaired, it would be helpful to roll WAL on a very slow sync to get a new pipeline.
      Otherwise the slow WAL will impact write latency for a long time (slow writes result in less writes result in the WAL not being rolled for longer)

      Attachments

        1. HBASE-21806.patch
          8 kB
          Sergey Shelukhin

        Issue Links

          relates to

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-22301 Consider rolling the WAL if the HDFS write pipeline is slow

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Activity

            People

              sershe Sergey Shelukhin
              sershe Sergey Shelukhin
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: