Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-26107

MOB compaction with missing files catches incorrect exception

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0-alpha-1
    • 3.0.0-alpha-2
    • mob
    • None

    Description

      The MOB compaction catches FileNotFoundException when hbase.unsafe.mob.discard.miss is true to handle missing MOB cells. The FNFE is wrapped in DoNotRetryIOException so the compaction fails for the given region.

      2021-07-21 13:51:05,880 WARN org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor: hbase.unsafe.mob.discard.miss=true. This is unsafe setting recommended only when first upgrading to a version with the distributed mob compaction feature on a cluster that has experienced MOB data corruption.
      2021-07-21 13:51:05,880 WARN org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor: hbase.unsafe.mob.discard.miss=true. This is unsafe setting recommended only when first upgrading to a version with the distributed mob compaction feature on a cluster that has experienced MOB data corruption.
      2021-07-21 13:51:05,880 INFO org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor: Compact MOB=true optimized configured=false optimized enabled=false maximum MOB file size=1073741824 major=true store=[table=IntegrationTestIngestWithMOB family=test_cf region=3a2ee81f9244c39ba61d694e616c1a89]
      2021-07-21 13:51:05,880 INFO org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor: Compact MOB=true optimized configured=false optimized enabled=false maximum MOB file size=1073741824 major=true store=[table=IntegrationTestIngestWithMOB family=test_cf region=7a96f55bb9ae04500a06cbaef02da6a3]
      2021-07-21 13:51:05,888 INFO org.apache.hadoop.hbase.regionserver.RSRpcServices: Compacting IntegrationTestIngestWithMOB,77777777,1626787996628.c71cad04514b17ee86a407490bd27424.
      2021-07-21 13:51:05,891 INFO org.apache.hadoop.hbase.regionserver.RSRpcServices: Compacting IntegrationTestIngestWithMOB,88888888,1626787996628.8fd002bda07755decda67b7084d1e0f6.
      2021-07-21 13:51:05,895 ERROR org.apache.hadoop.hbase.regionserver.HMobStore: The mob file 1bbd886460827015e5d605ed44252251202107200e5065290b424e38992f5556d9943b6a_7a96f55bb9ae04500a06cbaef02da6a3 could not be found in the locations [hdfs://example.com:8020/hbase/mobdir/data/default/IntegrationTestIngestWithMOB/e9b5d936e7f55a4f1c3246a8d5ce5
      3c2/test_cf, hdfs://example.com:8020/hbase/archive/data/default/IntegrationTestIngestWithMOB/e9b5d936e7f55a4f1c3246a8d5ce53c2/test_cf] or it is corrupt
      2021-07-21 13:51:05,895 INFO org.apache.hadoop.hbase.regionserver.throttle.PressureAwareThroughputController: 7a96f55bb9ae04500a06cbaef02da6a3#test_cf#compaction#1 average throughput is 0.07 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 10.00 MB/second
      2021-07-21 13:51:05,908 INFO org.apache.hadoop.hbase.regionserver.RSRpcServices: Compacting IntegrationTestIngestWithMOB,dddddddd,1626787996628.53186ca5008e3a964eee5f96ee3f1b26.
      2021-07-21 13:51:05,997 ERROR org.apache.hadoop.hbase.regionserver.CompactSplit: Compaction failed Request=regionName=IntegrationTestIngestWithMOB,11111111,1626787996628.7a96f55bb9ae04500a06cbaef02da6a3., storeName=test_cf, fileCount=1, fileSize=110.6 M (110.6 M), priority=1, time=1626875465819
      java.io.IOException: Mob compaction failed for region: 7a96f55bb9ae04500a06cbaef02da6a3
              at org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor.performCompaction(DefaultMobStoreCompactor.java:575)
              at org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:327)
              at org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor.compact(DefaultMobStoreCompactor.java:227)
              at org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:126)
              at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1407)
              at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2183)
              at org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.doCompaction(CompactSplit.java:633)
              at org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.run(CompactSplit.java:675)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: java.io.FileNotFoundException: File does not exist: hdfs://example.com:8020/hbase/archive/data/default/IntegrationTestIngestWithMOB/e9b5d936e7f55a4f1c3246a8d5ce53c2/test_cf/1bbd886460827015e5d605ed44252251202107200e5065290b424e38992f5556d9943b6a_7a96f55bb9ae04500a06cbaef02da6a3
              at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:438)
              at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:328)
              at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:307)
              at org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor.performCompaction(DefaultMobStoreCompactor.java:366)
              ... 10 more
      Caused by: java.io.FileNotFoundException: File does not exist: hdfs://example.com:8020/hbase/archive/data/default/IntegrationTestIngestWithMOB/e9b5d936e7f55a4f1c3246a8d5ce53c2/test_cf/1bbd886460827015e5d605ed44252251202107200e5065290b424e38992f5556d9943b6a_7a96f55bb9ae04500a06cbaef02da6a3
              at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1500)
              at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1493)
              at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
              at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1508)
              at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442)
              at org.apache.hadoop.hbase.regionserver.StoreFileInfo.<init>(StoreFileInfo.java:140)
              at org.apache.hadoop.hbase.regionserver.StoreFileInfo.<init>(StoreFileInfo.java:104)
              at org.apache.hadoop.hbase.regionserver.HStoreFile.<init>(HStoreFile.java:220)
              at org.apache.hadoop.hbase.mob.CachedMobFile.create(CachedMobFile.java:49)
              at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:220)
              at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:399)
              ... 13 more
      2021-07-21 13:51:06,013 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction of test_cf in IntegrationTestIngestWithMOB,77777777,1626787996628.c71cad04514b17ee86a407490bd27424.
      

      Attachments

        Issue Links

          Activity

            People

              psomogyi Peter Somogyi
              psomogyi Peter Somogyi
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: