Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
While testing HCatStorer with Iceberg realized Pig calls setStoreLocation on all Stores with the same Job object - https://github.com/apache/pig/blob/b050a33c66fc22d648370b5c6bda04e0e51d3aa3/src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java#L1081
Setting populated by one store is affecting the other stores. In my case the
"mapred.output.committer.class" is set as HiveIcebergCommitter by PigStore that is used by the Iceberg table and the other stores which inserts data to a non-iceberg tables also use that setting and trying to use HiveIcebergCommitter.
On checking with rohini , it is called to get the credentials from all stores since addCredentials API was added later and not all stores have implemented it and some still set configuration in setLocation method (i.e, HCatStorer).
Fixed it by passing a separate copy of Job object to each store's setLocation method and adding only the credential object from the call.
Attachments
Attachments
Issue Links
- links to