Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-16533

COMPOSITE_CRC failed between replicated file and striped file due to invalid requested length

    XMLWordPrintableJSON

Details

    Description

      After testing the COMPOSITE_CRC with some random length between replicated file and striped file which has same data with replicated file, it failed. 

      Reproduce step like this:

      @Test(timeout = 90000)
      public void testStripedAndReplicatedFileChecksum2() throws Exception {
        int abnormalSize = (dataBlocks * 2 - 2) * blockSize +
            (int) (blockSize * 0.5);
        prepareTestFiles(abnormalSize, new String[] {stripedFile1, replicatedFile});
      
        int loopNumber = 100;
        while (loopNumber-- > 0) {
          int verifyLength = ThreadLocalRandom.current()
              .nextInt(10, abnormalSize);
          FileChecksum stripedFileChecksum1 = getFileChecksum(stripedFile1,
              verifyLength, false);
          FileChecksum replicatedFileChecksum = getFileChecksum(replicatedFile,
              verifyLength, false);
      
          if (checksumCombineMode.equals(ChecksumCombineMode.COMPOSITE_CRC.name())) {
            Assert.assertEquals(stripedFileChecksum1, replicatedFileChecksum);
          } else {
            Assert.assertNotEquals(stripedFileChecksum1, replicatedFileChecksum);
          }
        }
      } 

      And after tracing the root cause, `FileChecksumHelper#makeCompositeCrcResult` maybe compute an error `consumedLastBlockLength` when updating checksum for the last block of the fixed length which maybe not the last block in the file.

      Attachments

        Issue Links

          Activity

            People

              xuzq_zander ZanderXu
              xuzq_zander ZanderXu
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 4h 50m
                  4h 50m