Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
2.4.3
-
None
-
None
Description
Structured streaming queries which involve more than one non-map-like transformations (either left joins or aggregations) produce incorrect (and sometimes inconsistent) results.
I have put up a sample here as Gist https://gist.github.com/jammann/b58bfbe0f4374b89ecea63c1e32c8f17
The test case involves 4 topics, and I've implement 6 different use cases of which 3 fail to produce the expected results
1) A innerjoin B: to topic A_B
expected: 10 inner joined messages
observation: OK
2) A innerjoin B outerjoin C: to topic A_B_outer_C
expected: 9 inner joined messages, last one in watermark
observation: OK
3) A innerjoin B outerjoin C outerjoin D: to topic A_B_outer_C_outer_D
expected: 9 inner/outer joined messages, 3 match C, 1 match D, last one in watermark
observation: NOK, only 3 messages are produced, where the C outer join matches
4) B outerjoin C: to topic B_outer_C
expected: 9 outer joined messages, 3 match C, 1 match D, last one in watermark
observation: NOK, same as 3
5) A innerjoin B agg on field of A: to topic A_inner_B_agg
expected: 3 groups with a total of 9 inner joined messages, last one in watermark
observation: OK
6) A innerjoin B outerjoin C agg on field of A: to topic A_inner_B_outer_C_agg
expected: 3 groups with a total of 9 inner joined messages, last one in watermark
observation: NOK, 3 groups with 1 message each, where C outer join matches
Attachments
Issue Links
- is duplicated by
-
SPARK-33259 Joining 3 streams results in incorrect output
- Resolved
- relates to
-
SPARK-28074 [SS] Log warn message on possible correctness issue for multiple stateful operations in single query
- Resolved