Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
TEZ-4229 handled the case of e.g. tez-local-cache folders and tez-conf.pb files, but seems like the config ("tez.local.cache.root.folder") has no effect on dag_*_priority.dot files.
I did the same hive unit test as on TEZ-4229 and I found that there are still lots of files there (that somehow I missed last time):
itests/hive-unit/dag_1612431668191_0001_10_priority.dot
itests/hive-unit/dag_1612431668191_0001_11_priority.dot
itests/hive-unit/dag_1612431668191_0001_12_priority.dot
itests/hive-unit/dag_1612431668191_0001_13_priority.dot
itests/hive-unit/dag_1612431668191_0001_14_priority.dot
itests/hive-unit/dag_1612431668191_0001_1_priority.dot
itests/hive-unit/dag_1612431668191_0001_2_priority.dot
itests/hive-unit/dag_1612431668191_0001_3_priority.dot
itests/hive-unit/dag_1612431668191_0001_4_priority.dot
itests/hive-unit/dag_1612431668191_0001_5_priority.dot
itests/hive-unit/dag_1612431668191_0001_6_priority.dot
itests/hive-unit/dag_1612431668191_0001_7_priority.dot
itests/hive-unit/dag_1612431668191_0001_8_priority.dot
sometimes I also catch another resources on watch git status, like:
itests/hive-unit/hive-exec-3.1.3000.7.1.7.0-SNAPSHOT-170a3a6a1dffe
UPDATE: it turned out that the problem is that Utils.generateDAGVizFile has a 3 parameters version without having to supply logDirs, and resolves to env variable. This is not sufficient as the user has to take care of that. With a simple change, we can propagate logDirs from DAGAppMaster through DagImpl to this codepath.
UPDATE: the root cause was TEZ-3958, which introduced a new method to Utils
