[FLINK-29766] AdaptiveBatchScheduler should also work with hybrid shuffle mode - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: 1.17.0
Fix Version/s: 1.17.0
Component/s: Runtime / Coordination
Labels:
- Umbrella

Description

AdaptiveBatchScheduler(including speculative execution) is a very important piece for large-scale batch jobs, but it can only work in the BLOCKING shuffle mode, I propose to also support this feature for HYBRID shuffle mode to combine the power of hybrid shuffle and adaptiveBatchScheduler.

Attachments

Issue Links

causes

FLINK-30938 Release Testing: Verify FLINK-29766 Adaptive Batch Scheduler should also work with hybrid shuffle mode

Closed

Sub-Tasks

1.	VertexWiseSchedulingStrategy supports hybrid shuffle	Closed	Weijie Guo
2.	Hybrid shuffle supports consume partial finished subtask in speculative execution mode	Closed	Weijie Guo
3.	Further limit the explosion range of failover in hybrid shuffle mode	Closed	Weijie Guo
4.	Hybrid full result partition should also work with speculative execution	Closed	Weijie Guo
5.	Partition finished state in ConsumedPartitionGroup is not updated correctly in dynamic graph	Closed	Weijie Guo

Activity

People

Assignee:: Weijie Guo

Reporter:: Weijie Guo

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Dates

Created:: 26/Oct/22 11:37

Updated:: 15/Feb/23 05:45

Resolved:: 15/Feb/23 05:45