Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-15570

Hadoop Erasure Coding ISA-L Check Request..

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Test
    • Status: Resolved
    • Trivial
    • Resolution: Fixed
    • 3.2.1
    • 3.2.1
    • None
    • -OS Version : CentOS Linux release 8.0.1905 (Core)

      -Hadoop Version : hadoop-3.2.1

       

    Description

      I am testing the performance of erasure coding in Hadoop 3.2.1 version environment.
      Apply ISA-L by referring to the manual (https://hadoop.apache.org/docs/r3.2.1/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html)

      ▶ JOB LIST (Proceed in manual order)

      1.  ISA-L library installed on all servers
        ld ISA-L library : https://github.com/01org/isa-l
      2.  hadoop build source (hadoop-3.2.1-src.tar.gz / mvn package -Pdist,native -Dtar -Drequire.isal -Drequire.snappy)
      3.  Deploy the built hadoop-3.2.1 folder to all servers

      But In Test (File upload Time/Select Orc table in Hive)
      There is no improvement in performance when tested.
      Ask for advice on work

      [Question 1]
      Is there anything wrong with the installation? Are there any missing tasks?
      [Question 2]
      Why doesn't it speed up ? (File upload Time/Select Orc table in Hive)

      [Question 3]
      Whether to use ISA-L in Hadoop, How to check?

       ※ When I used the "hdfs ec -listCodecs" command, I expected to see ISA-L. But no 

      ※ The warn log that occurs before applying ISA-L is It no longer occurs.
      ----------------------------------------------
      WARN org.apache.hadoop.io.erasurecode.ErasureCodeNative: ISA-L support is not available in your platform... using builtin-java codec where applicable
      ----------------------------------------------

       

      ▶Reference information

      =======================================

      -CPU INFO (namenode :2  / datanode : 5)
      ---------------------------------------------
      namenode cpu : Intel(R) Xeon(R) CPU E5-2609 v2 @ 2.50GHz
      datanode cpu : Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
      ---------------------------------------------

      -hadoop checknative (After source build)
      ---------------------------------------------
      Native library checking:
      hadoop: true /nds/hadoop-3.2.1/lib/native/libhadoop.so.1.0.0
      zlib: true /lib64/libz.so.1
      zstd : false
      snappy: true /lib64/libsnappy.so.1
      lz4: true revision:10301
      bzip2: false
      openssl: true /lib64/libcrypto.so
      ISA-L: true /lib64/libisal.so.2
      ---------------------------------------------

      -hdfs ec -listCodecs
      Erasure Coding Codecs: Codec [Coder List]
      RS [RS_NATIVE, RS_JAVA]
      RS-LEGACY [RS-LEGACY_JAVA]
      XOR [XOR_NATIVE, XOR_JAVA]

      =======================================

       

       

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            isurika isurika
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment