Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
1.6.0
-
None
-
None
Description
Testing Oozie through Knox proxy I found that there are some strange rewrite rule which seems outdated:
<detect path="name" value="inputDir">
<apply path="value" rule="OOZIE/oozie/hdfs-path"/>
</detect>
<detect path="name" value="outputDir">
<apply path="value" rule="OOZIE/oozie/hdfs-path"/>
</detect>
This ticket is intended to track the work removing them.
inputDir and outputDir are frequently used in Oozie's job.properties as a single directory name instead of a full HDFS path so in these cases the Oozie workflow fails running due to incorrect variable resolution:
Configuration in job.properties:
nameNode=WILL_BE_UPDATED_BY_KNOX outputDir=my_custom_output_dir
workflow.xml:
<prepare> <delete path="${nameNode}/user/${wf:user()}/examples/output-data/${outputDir}"/> </prepare>
Error in Oozie launcher:
Launcher AM execution failed java.lang.IllegalArgumentException: java.net.URISyntaxException: Expected scheme-specific part at index 5: hdfs: at org.apache.hadoop.fs.Path.initialize(Path.java:259) at org.apache.hadoop.fs.Path.<init>(Path.java:217) at org.apache.hadoop.fs.Path.<init>(Path.java:125) at org.apache.hadoop.fs.Globber.doGlob(Globber.java:285) at org.apache.hadoop.fs.Globber.glob(Globber.java:202) at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:2107) at org.apache.oozie.action.hadoop.FSLauncherURIHandler.delete(FSLauncherURIHandler.java:59) at org.apache.oozie.action.hadoop.PrepareActionsHandler.execute(PrepareActionsHandler.java:83) at org.apache.oozie.action.hadoop.PrepareActionsHandler.prepareAction(PrepareActionsHandler.java:74) at org.apache.oozie.action.hadoop.LauncherAM.executePrepare(LauncherAM.java:378) at org.apache.oozie.action.hadoop.LauncherAM.access$100(LauncherAM.java:55) at org.apache.oozie.action.hadoop.LauncherAM$2.run(LauncherAM.java:229) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899) at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:226) at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:156) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899) at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:144) Caused by: java.net.URISyntaxException: Expected scheme-specific part at index 5: hdfs: at java.net.URI$Parser.fail(URI.java:2847) at java.net.URI$Parser.failExpecting(URI.java:2853) at java.net.URI$Parser.parse(URI.java:3056) at java.net.URI.<init>(URI.java:746) at org.apache.hadoop.fs.Path.initialize(Path.java:256) ... 20 more
Found the real HDFS path after debugging the Oozie action:
hdfs://a.b.c.d:8020/user/test/examples/output-data/hdfs://a.b.c.d:8020/my_custom_output_dir
Attachments
Issue Links
- links to
