Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-36679

Remove lz4 hadoop wrapper classes after Hadoop 3.3.2

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 3.3.0
    • 3.3.0
    • Spark Core
    • None

    Description

      Lz4-java as provided dependency is not correctly excluded from relocation in Hadoop shaded client libraries in Hadoop 3.3.1. (HADOOP-17891)

       

      In order to deal the issue without reverting back to non-shade client libraries, we add a few Lz4 Hadoop wrapper classes `LZ4Factory`, `LZ4Compressor`, and `LZ4SafeDecompressor`, under the package `org.apache.hadoop.shaded.net.jpountz.lz4`.

      We should remove these wrapper classes after Hadoop 3.3.2 release which should include the fix.

       

      Attachments

        Issue Links

          depends upon

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-37600 Upgrade to Hadoop 3.3.2

          • Major - Major loss of function.
          • Resolved
          is duplicated by

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-37600 Upgrade to Hadoop 3.3.2

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-36669 Fail to load Lz4 codec

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Activity

            People

              Unassigned Unassigned
              viirya L. C. Hsieh
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: