Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-29426

Watermark does not take effect

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.4.3
    • Fix Version/s: None
    • Component/s: Structured Streaming
    • Labels:
      None

      Description

      I use withWatermark and window to express windowed aggregations, but the Watermark does not take effect.

      my code:

      
      // code placeholder
      Dataset<Row> clientSqlIpCount = mes.withWatermark("timestamp","1 minute")
              .groupBy(
                      functions.window(mes.col("timestamp"),"10 minutes","5 minutes"),
                      mes.col("sql"),mes.col("client"),mes.col("ip"))
              .count();
      StreamingQuery query = clientSqlIpCount
                      .writeStream()
                      .outputMode("Update")
                      .format("console")
                      .start();
      spark.streams().awaitAnyTermination();
      

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              jingshang jingshanglu
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: