Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-375

HadoopIO and runners-spark conflict with hadoop.version

Details

    Description

      HadoopIO currently uses 2.7.0 and runners-spark uses 2.2.0 for hadoop-client, hadoop-common.

      From amitsela
      "Spark can be built against different hadoop versions, but the release in maven central is a 2.2.0 build (latest). ''

      For HadoopIO, I don't know why 2.7.0 is picked at the beginning. I can check if it will work with 2.2.0.

      I am creating this issue, since I think it there is a general question.
      In principle, HadoopIO and other sdks Sources should work with any runners. But, when one set of runners require version A, but the other set of runners require version B, we will need a general solution for it.

      Attachments

        Activity

          People

            Unassigned Unassigned
            peihe0@gmail.com Pei He
            Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: