Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-46906

Add a check for stateful operator change for streaming

    XMLWordPrintableJSON

Details

    Description

      Currently user will get a misleading error as org.apache.spark.sql.execution.streaming.state.StateSchemaNotCompatible if restarting query in the same checkpoint location and changing their stateful operator. We need to catches such errors and throws a new error with informative message.

      After physical planning, before execution phase, we will read from state metadata with the current operator id to fetch operator name of committed batch with the same operator id. If operator name does not match, throws the error.

      Attachments

        Activity

          People

            jing.zhan Jing Zhan
            jing.zhan Jing Zhan
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: