Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-27519

Another case for FNFE on StoreFileScanner after a flush followed by a compaction

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.6.0, 3.0.0-alpha-3, 2.4.15, 2.5.2
    • 2.6.0, 3.0.0-alpha-4, 2.4.16, 2.5.3
    • None
    • None

    Description

      Besides the fix for HStore.getScanners for FNFE on StoreFileScanner after a flush followed by a compaction in HBASE-27484, there is a another case would cause FNFE:

      1. When StoreScanner is scanning, there is a flush followed by a compaction. When the flushed is completed in HStore.completeFlush and after it adds the new HStoreFile in StoreEngine, it would notify the StoreScanner.updateReaders by HStore.notifyChangedReadersObservers with the new flushed HStoreFile
            private boolean completeFlush(List<HStoreFile> sfs, long snapshotId) throws IOException {
            // NOTE:we should keep clearSnapshot method inside the write lock because clearSnapshot may
            // close {@link DefaultMemStore#snapshot}, which may be used by
            // {@link DefaultMemStore#getScanners}.
            storeEngine.addStoreFiles(sfs,
              snapshotId > 0 ? () -> this.memstore.clearSnapshot(snapshotId) : () -> {
              });
            // notify to be called here - only in case of flushes
            notifyChangedReadersObservers(sfs);
            if (LOG.isTraceEnabled()) {
              long totalSize = getTotalSize(sfs);
              String traceMessage = "FLUSH time,count,size,store size,store files ["
                + EnvironmentEdgeManager.currentTime() + "," + sfs.size() + "," + totalSize + ","
                + storeSize + "," + storeEngine.getStoreFileManager().getStorefileCount() + "]";
              LOG.trace(traceMessage);
            }
            return needsCompaction();
          }
        
      2. But if before entering StoreScanner.updateReaders , the compaction task starts and it compacts the new flushed HStoreFile and then HStore.closeAndArchiveCompactedFiles is called by CompactedHFilesDischarger to delete the HStoreFile.
      3. When StoreScanner.updateReaders continues to execute, it would read already deleted HStoreFile and throws FNFE.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            comnetwork chenglei
            comnetwork chenglei
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment