Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-14987

Compaction marker whose region name doesn't match current region's needs to be handled

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.2.0, 1.3.0, 0.98.17, 2.0.0
    • None
    • None
    • Reviewed

    Description

      One customer encountered the following error when replaying recovered edits, leading to region open failure:

      region=table1,d6b-2282-9223370590058224807-U-9856557-        EJ452727-16313786400171,1449616291799.fa8a526f2578eb3630bb08a4b1648f5d., starting to roll back the global memstore   size.
      org.apache.hadoop.hbase.regionserver.WrongRegionException: Compaction marker from WAL table_name: "table1"
      encoded_region_name: "d389c70fde9ec07971d0cfd20ef8f575"
      ...
      region_name: "table1,d6b-2282-9223370590058224807-U-9856557-EJ452727-16313786400171,1449089609367.d389c70fde9ec07971d0cfd20ef8f575."
       targetted for region d389c70fde9ec07971d0cfd20ef8f575 does not match this region: {ENCODED => fa8a526f2578eb3630bb08a4b1648f5d, NAME => 'table1,d6b-2282-                        9223370590058224807-U-9856557-EJ452727-16313786400171,1449616291799.fa8a526f2578eb3630bb08a4b1648f5d.', STARTKEY => 'd6b-2282-9223370590058224807-U-9856557-EJ452727-             16313786400171', ENDKEY => 'd76-2553-9223370588576178807-U-7416904-EK875822-17662180600000'}
        at org.apache.hadoop.hbase.regionserver.HRegion.checkTargetRegion(HRegion.java:4592)
        at org.apache.hadoop.hbase.regionserver.HRegion.replayWALCompactionMarker(HRegion.java:3831)
        at org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEdits(HRegion.java:3747)
        at org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEditsIfAny(HRegion.java:3601)
        at org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionStores(HRegion.java:911)
        at org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:789)
        at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:762)
        at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5774)
        at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5744)
      

      This was likely caused by the following action of hbck:

      15/12/08 18:11:34 INFO util.HBaseFsck: [hbasefsck-pool1-t37] Moving files from hdfs://Zealand/hbase/data/default/table1/d389c70fde9ec07971d0cfd20ef8f575/recovered.edits into     containing region hdfs://Zealand/hbase/data/default/table1/fa8a526f2578eb3630bb08a4b1648f5d/recovered.edits
      

      The recovered.edits for d389c70fde9ec07971d0cfd20ef8f575 contained compaction marker which couldn't be replayed against fa8a526f2578eb3630bb08a4b1648f5d

      Attachments

        1. 14987-suggest.txt
          3 kB
          Ted Yu
        2. 14987-v1.txt
          3 kB
          Ted Yu
        3. 14987-v2.txt
          16 kB
          Ted Yu
        4. 14987-v2.txt
          16 kB
          Ted Yu
        5. 14987-v3.txt
          9 kB
          Ted Yu
        6. 14987-v4.txt
          9 kB
          Ted Yu
        7. 14987-v5.txt
          9 kB
          Ted Yu

        Activity

          People

            yuzhihong@gmail.com Ted Yu
            yuzhihong@gmail.com Ted Yu
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: