Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
1.3.0
-
None
-
None
Description
HIVE-11540 solution is somewhat incomplete.
Suppose hive.compactor.max.num.delta=10 and there are 1000 delta files to compact.
CompactorMR will currently run 100 jobs on 10 files each to create 100 wider delta files but then it will run 1 job which will include 100 of the wider deltas.
it's a simple fix to make sure the max.num.delta is always respected
Attachments
Issue Links
- is blocked by
-
HIVE-11540 Too many delta files during Compaction - OOM
- Closed
- is related to
-
HIVE-14179 Too many delta files causes select queries on the table to fail with OOM
- Open