Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
Description
There are two paths for Spark set reducer parallelism, depending on the config hive.spark.use.op.stats. We currently handle all the logic in SetSparkReducerParallelism, and the logic is a bit complicated. Ideally we should refactor this to make it clearer, perhaps with separate rules for the paths.
Attachments
Issue Links
- is related to
-
HIVE-15796 HoS: poor reducer parallelism when operator stats are not accurate
- Resolved