Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-18764

add slow read block log entry to alert slow datanodeinfo when reading a block is slow

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 1.1.2
    • Fix Version/s: None
    • Component/s: HFile
    • Labels:
      None

      Description

      HBASE is on top of HDFS and both are distributed system. HBASE will also get impacted when there is struggler datanode due to network/disk/cpu issue. All HBASE read/scan towards that datanode will be slowdown. It's not easy for hbase admin to find out the struggler datanode in such case.

      While we have a log entry known as slow sync. One such entry is like the following. It will help hbase admin to quickly identify the slow datanode in the pipline in case of network/disk/cup issue with one of the 3 datanods in pipeline.

      2017-07-08 19:11:30,538 INFO  [sync.3] wal.FSHLog: Slow sync cost: 490189 ms, current pipeline: [DatanodeInfoWithStorage[xx.xx.xx.xx:50010,DS-c391299a-aa9f-4146-ac7e-a493ae536bff,DISK], DatanodeInfoWithSt
      orage[xx.xx.xx.xx:50010,DS-21a85f8b-f389-4f9e-95a8-b711945fd210,DISK], DatanodeInfoWithStorage[xx.xx.xx.xx:50010,DS-aa48cef2-3554-482f-b49d-be4763f4d8b8,DISK]]
      

      Inspired by slow sync log entry, I think it will also be beneficial for us to print out such kind of entry when we encounter slow read case. So that it will be easy to identify the slow datanode.

        Attachments

        1. HBASE-18764.rev1.1.2.patch
          4 kB
          Wang, Xinglong

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              suxingfate Wang, Xinglong
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated: