Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
Description
FSCountersSummary is only printed when LLAP IO is enabled, however, it's not LLAP IO specific as far as I know:
https://github.com/apache/hive/blob/03a76ac70370fb94a78b00496ec511e671c652f2/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/TezJobMonitor.java#L446-L449
it gives useful summary like:
INFO : Scheme: S3A INFO : ---------------------------------------------------------------------------------------------- INFO : VERTICES BYTES_READ READ_OPS LARGE_READ_OPS BYTES_WRITTEN WRITE_OPS INFO : ---------------------------------------------------------------------------------------------- INFO : Map 1 68.53KB 4 0 0B 0 INFO : Map 11 5.54MB 68 0 0B 0 INFO : Map 2 238.48MB 480 0 0B 0 INFO : Map 5 10.71MB 80 0 0B 0 INFO : Map 6 60.68MB 360 0 0B 0 INFO : Map 8 2.69MB 24 0 0B 0 INFO : Map 9 120.03MB 480 0 0B 0 INFO : Reducer 10 0B 0 0 0B 0 INFO : Reducer 3 0B 0 0 0B 0 INFO : Reducer 4 0B 0 0 0B 0 INFO : Reducer 7 0B 0 0 0B 0 INFO : ----------------------------------------------------------------------------------------------
need to check if it prints similar data correctly in case of Tez container mode as this is specific to the tez runner callable and the TaskCounterUpdater is called from tez codepath
Attachments
Issue Links
- links to