Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-8569

Support Hadoop 3 on Beam

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Triage Needed
    • P3
    • Resolution: Fixed
    • None
    • 2.27.0
    • io-java-hadoop-file-system
    • None

    Description

      It seems that Hadoop 3 in production is finally happening. CDH supports it in their latest version and Spark 3 will include support for Hadoop 3 too.

      This is an uber ticket to cover the required changes to the codebase to ensure compliance with Hadoop 3.x

      Hadoop dependencies in Beam are mostly provided and APIs are until some point compatible, but we may require changes in the CI to test that new changes work both in Hadoop 2 and Hadoop 3 until we decide to remove support for Hadoop 2.

      Attachments

        Issue Links

          is related to

          New Feature - A new feature of the product, which has yet to be developed. BEAM-7093 Support Spark 3 in Spark runner

          • P3 - Non-urgent bugs, features, and improvements
          • Triage Needed

          Improvement - An improvement or enhancement to an existing feature or task. BEAM-4087 Gradle build does not allow to overwrite versions of provided dependencies

          • P3 - Non-urgent bugs, features, and improvements
          • Open
          requires

          Test - A new unit, integration or system test. BEAM-8876 Tests to confirm Hadoop version compatibility

          • P3 - Non-urgent bugs, features, and improvements
          • Resolved
          links to

          Web Link GitHub Pull Request #13354

          Activity

            People

              piotr-szuberski Piotr Szuberski
              iemejia Ismaël Mejía
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 4h 50m
                  4h 50m