Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-15759

EC: Verify EC reconstruction correctness on DataNode

    XMLWordPrintableJSON

Details

    Description

      EC reconstruction on DataNode has caused data corruption: HDFS-14768, HDFS-15186 and HDFS-15240. Those issues occur under specific conditions and the corruption is neither detected nor auto-healed by HDFS. It is obviously hard for users to monitor data integrity by themselves, and even if they find corrupted data, it is difficult or sometimes impossible to recover them.

      To prevent further data corruption issues, this feature proposes a simple and effective way to verify EC reconstruction correctness on DataNode at each reconstruction process.
      It verifies correctness of outputs decoded from inputs as follows:
      1. Decoding an input with the outputs;
      2. Compare the decoded input with the original input.
      For instance, in RS-6-3, assume that outputs [d1, p1] are decoded from inputs [d0, d2, d3, d4, d5, p0]. Then the verification is done by decoding d0 from [d1, d2, d3, d4, d5, p1], and comparing the original and decoded data of d0.

      When an EC reconstruction task goes wrong, the comparison will fail with high probability.
      Then the task will also fail and be retried by NameNode.
      The next reconstruction will succeed if the condition triggered the failure is gone.

      Attachments

        Activity

          People

            touchida Toshihiko Uchida
            touchida Toshihiko Uchida
            Votes:
            0 Vote for this issue
            Watchers:
            16 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 10h 20m
                10h 20m