Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14476

lock too long when fix inconsistent blocks between disk and in-memory

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.6.0, 2.7.0, 3.0.3
    • 3.3.0, 2.10.1
    • datanode
    • None

    Description

      When directoryScanner have the results of differences between disk and in-memory blocks. it will try to run checkAndUpdate to fix it. However FsDatasetImpl.checkAndUpdate is a synchronized call

      As I have about 6millions blocks for every datanodes and every 6hours' scan will have about 25000 abnormal blocks to fix. That leads to a long lock holding FsDatasetImpl object.

      let's assume every block need 10ms to fix(because of latency of SAS disk), that will cost 250 seconds to finish. That means all reads and writes will be blocked for 3mins for that datanode.

       

      2019-05-06 08:06:51,704 INFO org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: BlockPool BP-1644920766-10.223.143.220-1450099987967 Total blocks: 6850197, missing metadata files:23574, missing block files:23574, missing blocks in memory:47625, mismatched blocks:0
      
      ...
      
      2019-05-06 08:16:41,625 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Took 588402ms to process 1 commands from NN
      

      Take long time to process command from nn because threads are blocked. And namenode will see long lastContact time for this datanode.

      Maybe this affect all hdfs versions.

      how to fix:

      just like process invalidate command from namenode with 1000 batch size, fix these abnormal block should be handled with batch too and sleep 2 seconds between the batch to allow normal reading/writing blocks.

      Attachments

        1. datanode-with-patch-14476.png
          93 kB
          Sean Chow
        2. HDFS-14476.00.patch
          2 kB
          Sean Chow
        3. HDFS-14476.002.patch
          2 kB
          Wei-Chiu Chuang
        4. HDFS-14476.01.patch
          2 kB
          Sean Chow
        5. HDFS-14476.branch-3.2.001.patch
          2 kB
          Wei-Chiu Chuang
        6. HDFS-14476-branch-2.01.patch
          2 kB
          Sean Chow
        7. HDFS-14476-branch-2.02.patch
          3 kB
          Sean Chow
        8. HDFS-14476-branch-2.10.02.patch
          3 kB
          Ayush Saxena

        Issue Links

          Activity

            People

              seanlook Sean Chow
              seanlook Sean Chow
              Votes:
              0 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: