Details
-
Improvement
-
Status: Resolved
-
Minor
-
Resolution: Fixed
-
3.0.0
-
None
Description
Please refer https://lists.apache.org/thread.html/cc6489a19316e7382661d305fabd8c21915e5faf6a928b4869ac2b4a@%3Cdev.spark.apache.org%3E to see rationalization of the issue.
tl;dr. Using multiple stateful operators without fully understanding how global watermark affects the query brings correctness issue.
While we may not want to fully block the operations given some users could leverage it with understanding of global watermark, it would be necessary for end users to warn on such breaking-correctness possibility, and provide alternative(s).
Attachments
Issue Links
- is related to
-
SPARK-28094 Multiple left joins or aggregations in one query produce incorrect results
- Open
- links to