Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-17045

File renamed from a snapshottable dir to a non-snapshottable dir cannot be deleted.

    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      HDFS-16972 added a shouldDestroy method which caused the following bug.

      Background:

      • When FileSystem.rename(src, dst) from a snapshottable dir (src) to a snapshottable dir (dst), dstSnapshotId is set to the latest snapshot at dst. As a result, dst is NOT in dstSnapshotId because dstSnapshotId was already taken before rename.
      • snapshotToBeDeleted is the snapshot id of the current operation if the operation is FileSystem.deleteSnapshot. Otherwise, snapshotToBeDeleted is set to CURRENT_STATE_ID.
      • If (snapshotToBeDeleted > dstSnapshotId), dst is in snapshotToBeDeleted. The shouldDestroy method returns true to continue deletion.
      • If (snapshotToBeDeleted <= dstSnapshotId), dst must not be in snapshotToBeDeleted. The shouldDestroy method returns false to stop deletion.

      All the above are correct for renaming within snapshottable directories.

      Bug:

      • If rename(src, dst) from a snapshottable dir (src) to a non-snapshottable dir (dst), dstSnapshotId becomes CURRENT_STATE_ID.
      • When FileSystem.delete(dst) happens, snapshotToBeDeleted is also set to CURRENT_STATE_ID.
      • In this case, snapshotToBeDeleted == dstSnapshotId, the shouldDestroy method will return false and it incorrectly stops the deletion.

      Not that this bug may cause fsimage corruption and quota miscalculation since some files can be partially deleted. Fortunately, this bug won't cause data loss.

      Attachments

        Issue Links

          Activity

            People

              szetszwo Tsz-wo Sze
              szetszwo Tsz-wo Sze
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: