Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Incomplete
-
2.4.3
-
None
Description
I use withWatermark and window to express windowed aggregations, but the Watermark does not take effect.
my code:
// code placeholder Dataset<Row> clientSqlIpCount = mes.withWatermark("timestamp","1 minute") .groupBy( functions.window(mes.col("timestamp"),"10 minutes","5 minutes"), mes.col("sql"),mes.col("client"),mes.col("ip")) .count(); StreamingQuery query = clientSqlIpCount .writeStream() .outputMode("Update") .format("console") .start(); spark.streams().awaitAnyTermination();