Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-20331 clean up shaded packaging for 2.1
  3. HBASE-20332

shaded mapreduce module shouldn't include hadoop

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 2.0.0
    • 3.0.0-alpha-1, 2.1.0
    • mapreduce, shading
    • None
    • Hide
      <!-- markdown -->

      The `hbase-shaded-mapreduce` artifact no longer include its own copy of Hadoop classes. Users who make use of the artifact via YARN should be able to get these classes from YARN's classpath without having to make any changes.
      Show
      <!-- markdown --> The `hbase-shaded-mapreduce` artifact no longer include its own copy of Hadoop classes. Users who make use of the artifact via YARN should be able to get these classes from YARN's classpath without having to make any changes.

    Description

      AFAICT, we should just entirely skip including hadoop in our shaded mapreduce module

      1) Folks expect to run yarn / mr apps via hadoop jar / yarn jar
      2) those commands include all the needed Hadoop jars in your classpath by default (both client side and in the containers)
      3) If you try to use "user classpath first" for your job as a workaround (e.g. for some library your application needs that hadoop provides) then our inclusion of some but not all hadoop classes then causes everything to fall over because of mixing rewritten and non-rewritten hadoop classes
      4) if you don't use "user classpath first" then all of our non-relocated-but-still-shaded hadoop classes are ignored anyways so we're just wasting space

      Attachments

        1. HBASE-20332.7.patch
          70 kB
          Sean Busbey
        2. HBASE-20332.6.patch
          69 kB
          Sean Busbey
        3. HBASE-20332.5.patch
          68 kB
          Sean Busbey
        4. HBASE-20332.4.patch
          68 kB
          Sean Busbey
        5. HBASE-20332.3.patch
          102 kB
          Sean Busbey
        6. HBASE-20332.2.WIP.patch
          85 kB
          Sean Busbey
        7. HBASE-20332.1.WIP.patch
          82 kB
          Sean Busbey
        8. HBASE-20332.0.patch
          37 kB
          Sean Busbey

        Issue Links

          Activity

            People

              busbey Sean Busbey
              busbey Sean Busbey
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: