Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
When tracking DAG progress, TezClientUtils ends up creating new remote user. Because of this new UGI creation, IPC connections are not reused internally.
More info from Hadoop side:
In hadoop's IPC layer, connectionIds are checked based on UserGroupInformation.
https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java#L1600
However, UserGroupInformation comparison is based on ==
https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java#L1789