Details
-
New Feature
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
Description
The past decade has witnessed a dramatic shift in Flink's deployment mode, workload patterns, and hardware improvements. We've moved from the map-reduce era where workers are computation-storage tightly coupled nodes to a cloud-native world where containerized deployments on Kubernetes become standard. To enable Flink's Cloud-Native future, we introduce Disaggregated State Storage and Management that uses DFS as primary storage in Flink 2.0
This new architecture is aimed to solve the following challenges brought in the cloud-native era for Flink.
1. Local Disk Constraints in containerization
2. Spiky Resource Usage caused by compaction in the current state model
3. Fast Rescaling for jobs with large states (hundreds of Terabytes)
4. Light and Fast Checkpoint in a native way
Design Details can be found in FLIP-423
Proposed changes can be found here:
Attachments
Issue Links
- is a parent of
-
-
- Open
-
-
-
- Open
-
-
-
- Open
-
-
-
- In Progress
-
-
-
- In Progress
-