Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-20724 Sometimes some compacted storefiles are still opened after region failover
  3. HBASE-22330

Backport HBASE-20724 (Sometimes some compacted storefiles are still opened after region failover) to branch-1

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.5.0, 1.4.9, 1.3.4
    • 1.4.10, 1.3.5
    • Compaction, regionserver
    • None
    • Reviewed

    Description

      There appears to be a race condition between close and split which when combined with a side effect of HBASE-20704, leads to the parent region store files getting archived and cleared while daughter regions still have references to those parent region store files.

      Here is the timeline of events observed for an affected region:

      1. RS1 faces ZooKeeper connectivity issue for master node and starts shutting itself down. As part of this it starts to close the store and clean up the compacted files (File A)
      2. Master starts bulk assigning regions and assign parent region to RS2
      3. Region opens on RS2 and ends up opening compacted store file(s) (suspect this is due to HBASE-20724)
      4. Now split happens and daughter regions open on RS2 and try to run a compaction as part of post open
      5. Split request at this point is complete. However now archiving proceeds on RS1 and ends up archiving the store file that is referenced by the daughter.

      Compaction fails due to FileNotFoundException and all subsequent attempts to open the region will fail until manual resolution.

      We think having HBASE-20724 would help in such situations since we won't end up loading compacted store files in the first place. 

      Attachments

        1. HBASE-22330.branch-1.001.patch
          61 kB
          Abhishek Singh Chouhan
        2. HBASE-22330.branch-1.002.patch
          62 kB
          Abhishek Singh Chouhan
        3. HBASE-22330.branch-1.3.001.patch
          62 kB
          Abhishek Singh Chouhan
        4. HBASE-22330-addendum.branch-1.patch
          2 kB
          Abhishek Singh Chouhan

        Issue Links

          Activity

            People

              abhishek.chouhan Abhishek Singh Chouhan
              apurtell Andrew Kyle Purtell
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: