Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-35959

Add a new Maven profile "no-shaded-client" for older Hadoop 3.x versions

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Won't Fix
    • 3.2.0
    • None
    • Build
    • None

    Description

      Currently Spark uses Hadoop shaded client by default. However, if Spark users want to build Spark with older version of Hadoop, such as 3.1.x, the shaded client cannot be used (currently it only support Hadoop 3.2.2+ and 3.3.1+). Therefore, this proposes to offer a new Maven profile "no-shaded-client" for this use case.

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-37994 Unable to build spark3.2 with -Dhadoop.version=3.1.4

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-33212 Upgrade to Hadoop 3.2.2 and move to shaded clients for Hadoop 3.x profile

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #33160 [Github] Pull Request #33160 (sunchao)

          Pull request #33160 [Github] Pull Request #33160 (sunchao)

          Activity

            People

              Unassigned Unassigned
              csun Chao Sun
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: