Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
0.6.0
-
None
-
None
Description
The use case is:
dir = hdfs://host:9000/user/warehouse/tableName/abc
pathToPartitionInfo = {/user/warehouse/tableName : myPart}
Then calling
getPartitionDescFromPath(dir, pathToPartitionInfo)
will throw an IOException because /user/warehouse/tableName is not a prefix of hdfs://host:9000/user/warehouse/tableName/abc. Currently, this is not an issue but will come up if CombineFileInputFormat is modified so what the scheme and authority are not stripped out when generating splits (see MAPREDUCE-1806).
The proposed solution is add a case where matching is done by just the path component of the URI's.