Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-2270

Examples archetype bundles Hadoop 2.6 in its jar for ApexRunner; cannot run on Hadoop 2.7?

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • P2
    • Resolution: Fixed
    • None
    • 2.3.0
    • examples-java, runner-apex
    • None

    Description

      In an instantiated examples archetype, with -P apex-runner, Apex depends on Hadoop 2.6.0 and this is bundles into the examples jar.

      In order to get this to run on Hadoop 2.7.3 I added this to the profile:

            <properties>
        <hadoop.version>2.7.3</hadoop.version>
      </properties>
 
      <dependencies>
        <dependency>
          <groupId>org.apache.hadoop</groupId>
          <artifactId>hadoop-yarn-client</artifactId>
          <version>${hadoop.version}</version>
        </dependency>
        <dependency>
          <groupId>org.apache.hadoop</groupId>
          <artifactId>hadoop-common</artifactId>
          <version>${hadoop.version}</version>
        </dependency>
      </dependencies>

      It is not clear to me what the best path is, here. Clearly the way we bundle is brittle and probably not the recommended best practice. But also perhaps the deps of the runner can be modified to provided.

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. BEAM-3240 Quickstart examples archetype dependencies conflict for Apex on YARN

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              kenn Kenneth Knowles
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: