Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-28042

Snapshot corruptions due to non-atomic rename within same filesystem

    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      Sequence of events that can lead to snapshot corruptions:

      1. Create snapshot using admin command
      2. Active master triggers async snapshot creation
      3. If the snapshot operation doesn't complete within 5 min, client gets exception
      org.apache.hadoop.hbase.snapshot.SnapshotCreationException: Snapshot 'T1_1691888405683_1691888440827_1' wasn't completed in expectedTime:600000 ms  
      1. Client initiates snapshot deletion after this error
      2. In the snapshot completion/commit phase, the files are moved from tmp to final dir.
      3. Snapshot delete and snapshot commit operations can cause corruption by leaving incomplete metadata:
      • [Snapshot commit] create "/hbase/.hbase-snapshot/T1_1691888405683_1691888440827_1/.snapshotinfo"
      • [Snapshot delete from client]  delete "/hbase/.hbase-snapshot/T1_1691888405683_1691888440827_1/.snapshotinfo"
      • [Snapshot commit]  create "/hbase/.hbase-snapshot/T1_1691888405683_1691888440827_1/data-manifest"

       

      The changes introduced by HBASE-21098 performs atomic rename for hbase 1 but not for hbase 2

        public static void completeSnapshot(Path snapshotDir, Path workingDir, FileSystem fs,
    FileSystem workingDirFs, final Configuration conf)
    throws SnapshotCreationException, IOException {
    LOG.debug(
      "Sentinel is done, just moving the snapshot from " + workingDir + " to " + snapshotDir);
    URI workingURI = workingDirFs.getUri();
    URI rootURI = fs.getUri();
    if (
      (!workingURI.getScheme().equals(rootURI.getScheme()) || workingURI.getAuthority() == null
        || !workingURI.getAuthority().equals(rootURI.getAuthority())
        || workingURI.getUserInfo() == null //always true for hdfs://{cluster}
        || !workingURI.getUserInfo().equals(rootURI.getUserInfo())
        || !fs.rename(workingDir, snapshotDir)) //this condition isn't even evaluated due to short circuit above
        && !FileUtil.copy(workingDirFs, workingDir, fs, snapshotDir, true, true, conf) // non-atomic rename operation
    ) {
      throw new SnapshotCreationException("Failed to copy working directory(" + workingDir
        + ") to completed directory(" + snapshotDir + ").");
    }
  }

      whereas for hbase 1

      // check UGI/userInfo
if (workingURI.getUserInfo() == null && rootURI.getUserInfo() != null) {
  return true;
}
if (workingURI.getUserInfo() != null &&
    !workingURI.getUserInfo().equals(rootURI.getUserInfo())) {
  return true;
}

      this causes shouldSkipRenameSnapshotDirectories() to return false if workingURI and rootURI share the same filesystem, which would always lead to atomic rename:

      if ((shouldSkipRenameSnapshotDirectories(workingURI, rootURI)
    || !fs.rename(workingDir, snapshotDir))
     && !FileUtil.copy(workingDirFs, workingDir, fs, snapshotDir, true, true, conf)) {
  throw new SnapshotCreationException("Failed to copy working directory(" + workingDir
      + ") to completed directory(" + snapshotDir + ").");
}

       

      Attachments

        Issue Links

          breaks

          Bug - A problem which impairs or prevents the functions of the product. HBASE-28081 Snapshot working dir does not retain ACLs after snapshot commit phase

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved
          links to

          Web Link GitHub Pull Request #5369

          Web Link GitHub Pull Request #5372

          Activity

            People

              vjasani Viraj Jasani
              vjasani Viraj Jasani
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: