Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-3576 [Umbrella] Make Oozie cloud ready
  3. OOZIE-3575

Add credential support for cloud file systems

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 5.2.0
    • 5.3.0
    • core
    • None

    Description

      Oozie by default gathers delegation tokens for the nodes defined in mapreduce.job.hdfs-servers (or oozie.launcher.mapreduce.job.hdfs-servers in case of distcp actions) and for the workflow path.

      Though this implementation is good for hdfs, we dont support occasions where the job related resources, which we want to access in runtime are present on different file systems/buckets etc...

      The HDFSCredentials class should be revised to handle getting tokens for different cloud storages.

      The following scenarios should be addressed:
      Oozie should obtain delegation token in case

      • the defaultFs is cloud
      • the workload.xml is in cloud
      • input/output/auxiliary files referred from workflow are in cloud
      • (newly introduced feature) user could define filesystem credentials for the workflow (as its done with hive/hcat etc..) -> this would allow the user to handle the situation where Oozie could not decide which tokens are needed at launch time by default and could also get tokens for different cloud storages and buckets as well

      Example for credentials addition:

      <credential name="aws_auth" type="filesystem">
        <property>
          <name>filesystem.path</name>
          <value>s3a://qe-s3-bucket-mst</value>
        </property>
      </credential>
      

      Attachments

        1. OOZIE-3575-v3.patch
          27 kB
          Mate Juhasz
        2. OOZIE-3575-v2.patch
          27 kB
          Mate Juhasz
        3. OOZIE-3575.patch
          24 kB
          Mate Juhasz

        Activity

          People

            matijhs Mate Juhasz
            matijhs Mate Juhasz
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: