Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-43197

Clean up the code written for compatibility with Hadoop 2

    XMLWordPrintableJSON

Details

    • Umbrella
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.5.0
    • None
    • Spark Core, SQL, YARN
    • None

    Description

      SPARK-42452 removed support for Hadoop2, we can clean up the code written for compatibility with Hadoop 2 to make it more concise

      Attachments

        1.
        		 Replace reflection w/ direct calling for `ContainerLaunchContext#setTokensConf` Sub-task Resolved Yang Jie
        2.
        		 Remove unnecessary serializable wrapper in HadoopFSUtils Sub-task Resolved Cheng Pan
        3.
        		 Remove workaround for HADOOP-12074 Sub-task Resolved Cheng Pan
        4.
        		 Replace reflection w/ direct calling for Hadoop CallerContext Sub-task Resolved Cheng Pan
        5.
        		 Remove workaround for MiniKdc's BindException Sub-task Resolved Cheng Pan
        6.
        		 Remove workaround for FileSinkDesc Sub-task Resolved Cheng Pan
        7.
        		 Inline `hadoop-client` related properties in `pom.xml` Sub-task Resolved Yang Jie
        8.
        		 Resume using enumeration to compare `NodeState.DECOMMISSIONING` Sub-task Resolved Yang Jie
        9.
        		 Remove Hadoop 2 reference in docs Sub-task Resolved Cheng Pan
        10.
        		 Replace reflection w/ direct calling for YARN Resource API Sub-task Resolved Cheng Pan
        11.
        		 Remove Hadoop2 support in IsolatedClientLoader Sub-task Resolved Cheng Pan
        12.
        		 Remove `ResourceRequestHelper#isYarnResourceTypesAvailable` Sub-task Resolved Unassigned
        13.
        		 Refactor `ResourceRequestHelper ` to no longer use reflection Sub-task Resolved Unassigned
        14.
        		 Remove check of `isHadoop3` Sub-task Resolved Yang Jie
        15.
        		 Unnecessary serialize/deserialize of Path on parallel gather partition stats Sub-task Resolved Cheng Pan
        16.
        		 Replace reflection w/ direct calling for `SparkHadoopUtil#createFile` Sub-task Resolved Yang Jie
        17.
        		 Clean up deprecation hadoop api usage in Yarn module Sub-task Resolved Yang Jie
        18.
        		 Directly call `replicate()` instead of reflection in `SparkHadoopUtil#createFile` Sub-task Resolved Yang Jie
        19.
        		 Replace the link related to hadoop version 2 with hadoop version 3 Sub-task Resolved BingKun Pan
        20.
        		 Organize hadoop-cloud in standard maven project structure Sub-task Resolved Cheng Pan

        Activity

          People

            Unassigned Unassigned
            LuciferYang Yang Jie
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated: