Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-16601

Add support for hardware crc32 of nativetask checksums on aarch64 arch

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Patch Available
    • Major
    • Resolution: Unresolved
    • 3.2.1
    • site
    • common
    • Patch

    Description

      Add support for aarch64 CRC instructions in nativetask module, optimize the CRC32 and CRC32C.

      Use the benchmark tools : nttest , the improvement is quite substantial:  

      CRC32 Zlib polynomial 0x04C11DB7

      KeyValueType-IO Before(MB/s) After(MB/s) Improvement
      TextType-Write 425.98 602.92 +42%
      TextType-Read 796.06 1716.59 +116%
      BytesType-Write 474.25 686.84 +45%
      BytesType-Read 844.96 1955.03 +131%
      UnknownType-Write 434.84 608.81 +40%
      UnknownType-Read 805.76 1733.82 +115%

       
       
      CRC32C  Castagnoli polynomial 0x1EDC6F41
       

      KeyValueType-IO Before(MB/s) After(MB/s) Improvement
      TextType-Write 423.39 606.55 +43%
      TextType-Read 799.20 1783.28 +123%
      BytesType-Write 473.95 696.47 +47%
      BytesType-Read 846.30 2018.06 +138%
      UnknownType-Write 434.07 612.31 +41%
      UnknownType-Read 807.16 1783.95 +121%

      Attachments

        1. aarch64_nativetask_test.log
          32 kB
          MacChen01
        2. aarch64_nativetask_test.log
          32 kB
          MacChen01
        3. HADOOP-16601.patch
          12 kB
          MacChen01

        Activity

          People

            Unassigned Unassigned
            MacChen01 MacChen01
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated: