[FLINK-25318] Improvement of scheduler and execution for Flink OLAP - ASF JIRA

Attach files

Attach Screenshot

Add vote

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Open
Priority: Major
Resolution: Unresolved
Affects Version/s: 1.14.0, 1.12.5, 1.13.3
Fix Version/s: None
Component/s: Runtime / Coordination, Runtime / Network
Labels:
- Umbrella

Description

We use flink to perform OLAP queries. We launch flink session cluster, submit batch jobs to the cluster as OLAP queries, and fetch the jobs' results. OLAP jobs are generally small queries which will finish at the seconds or milliseconds, and users always submit multiple jobs to the session cluster concurrently. We found the qps and latency of jobs will be greatly affected when there're tens jobs are running, even when there's little data in each query. We will give the result of benchmark for the latest version later.
After discussed with Xintong Song, and thanks for his advice, we create this issue to trace and manager Flink OLAP related improvements. More users and developers are welcome and feel free to create Flink OLAP related subtasks here, thanks

Attachments

Issue Links

Add Link

is a parent of

FLINK-25027 Allow GC of a finished job's JobMaster before the slot timeout is reached

Open

Delete this link

FLINK-33852 FLIP-403 High Availability Services for OLAP Scenarios

Open

Delete this link

relates to

FLINK-32898 Improvement of usability for Flink OLAP

Open

Delete this link

FLINK-33683 FLIP-407 Improve Flink Client performance in interactive scenarios

Open

Delete this link

FLINK-25356 Add benchmarks for performance in OLAP scenarios

Open

Delete this link

Sub-Tasks

Create Sub-Task

1.	System classloader memory leak after loading too many codegen classes.	Open	Unassigned	Actions
2.	Improvement of reuse segments for join/agg/sort operators in TaskManager for flink olap queries	Open	Unassigned	Actions
3.	Manage and share gateways of taskmanagers between jobs in session cluster	Open	Unassigned	Actions
4.	HiveSourceFileEnumerator should fetch splits asynchronously	Open	Unassigned	Actions
5.	Improvement of connection from TM to JM in session cluster	Open	Unassigned	Actions
6.	Too many JM logs in flink session cluster for olap queries	Open	Unassigned	Actions
7.	ExecutionGraphInfoStore in session cluster should split failed and successful jobs	Open	Zhanghao Chen	Actions
8.	Improve cache hit rate of generated class	Open	Dan Zou	Actions
9.	Add WebSocket in Dispatcher to support olap query submission and push results in session cluster	Open	Unassigned	Actions
10.	Improve to reuse threads in TaskManager for different tasks between jobs	Open	Unassigned	Actions
11.	Support customized listener during task manager startup	Open	Unassigned	Actions
12.	Support concurrency control when submitting OLAP jobs to Dispatcher	Open	Unassigned	Actions
13.	Use default classloader in TaskManager when there are no user jars for job	Open	Dan Zou	Actions