[HBASE-25206] Data loss can happen if a cloned table loses original split region(delete table) - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 3.0.0-alpha-1, 2.3.3, 2.4.0, 2.2.7
Component/s: proc-v2, Region Assignment, snapshots
Labels:
None

Hadoop Flags:

Reviewed

Description

Steps to reproduce are as follows:

1. Create a table and put some data into the table:

create 'test1','cf'
put 'test1','r1','cf','v1'
put 'test1','r2','cf','v2'
put 'test1','r3','cf','v3'
put 'test1','r4','cf','v4'
put 'test1','r5','cf','v5'

2. Take a snapshot for the table:

snapshot 'test1','snap_test'

3. Clone the snapshot to another table

clone_snapshot 'snap_test','test2'

4. Split the original table

split 'test1','r3'

5. Drop the original table

disable 'test1'
drop 'test1'

After that, we see the error like the following in RS log when opening the regions of the cloned table:

2020-10-20 13:32:18,415 WARN org.apache.hadoop.hbase.regionserver.HRegion: Failed initialize of region= test2,,1603200595702.bebdc4f740626206eeccad96b7643261., starting to roll back memstore
java.io.IOException: java.io.IOException: java.io.FileNotFoundException: Unable to open link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs://<NN HOST>:8020/hbase/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89, hdfs://<NN HOST>:8020/hbase/.tmp/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89, hdfs://<NN HOST>:8020/hbase/mobdir/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89, hdfs://<NN HOST>:8020/hbase/archive/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89]
        at org.apache.hadoop.hbase.regionserver.HRegion.initializeStores(HRegion.java:1095)
        at org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:943)
        at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:899)
        at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7246)
        at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7204)
        at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7176)
        at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7134)
        at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7085)
        at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:283)
        at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
        at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: java.io.FileNotFoundException: Unable to open link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs://<NN HOST>:8020/hbase/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89, hdfs://<NN HOST>:8020/hbase/.tmp/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89, hdfs://<NN HOST>:8020/hbase/mobdir/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89, hdfs://<NN HOST>:8020/hbase/archive/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89]
        at org.apache.hadoop.hbase.regionserver.HStore.openStoreFiles(HStore.java:590)
        at org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(HStore.java:557)
        at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:303)
        at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:5731)
        at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1059)
        at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1056)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        ... 3 more
Caused by: java.io.FileNotFoundException: Unable to open link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs://<NN HOST>:8020/hbase/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89, hdfs://<NN HOST>:8020/hbase/.tmp/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89, hdfs://<NN HOST>:8020/hbase/mobdir/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89, hdfs://<NN HOST>:8020/hbase/archive/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89]
        at org.apache.hadoop.hbase.io.FileLink$FileLinkInputStream.tryOpen(FileLink.java:322)
        at org.apache.hadoop.hbase.io.FileLink$FileLinkInputStream.<init>(FileLink.java:126)
        at org.apache.hadoop.hbase.io.FileLink$FileLinkInputStream.<init>(FileLink.java:117)
        at org.apache.hadoop.hbase.io.FileLink.open(FileLink.java:429)
        at org.apache.hadoop.hbase.io.FSDataInputStreamWrapper.<init>(FSDataInputStreamWrapper.java:125)
        at org.apache.hadoop.hbase.io.FSDataInputStreamWrapper.<init>(FSDataInputStreamWrapper.java:107)
        at org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:261)
        at org.apache.hadoop.hbase.regionserver.HStoreFile.open(HStoreFile.java:368)
        at org.apache.hadoop.hbase.regionserver.HStoreFile.initReader(HStoreFile.java:476)
        at org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:703)
        at org.apache.hadoop.hbase.regionserver.HStore.lambda$openStoreFiles$1(HStore.java:573)
        ... 6 more

Attachments

Issue Links

is blocked by

HBASE-25207 Revisit the implementation and usage of RegionStates.include

Closed

links to

GitHub Pull Request #2569

Activity

People

Assignee:: Toshihiro Suzuki

Reporter:: Toshihiro Suzuki

Votes:: 0 Vote for this issue

Watchers:: 8 Start watching this issue

Dates

Created:: 20/Oct/20 13:18

Updated:: 17/Jun/22 18:04

Resolved:: 24/Oct/20 15:22