Details
-
Task
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
4.0.0
Description
Currently user will get a misleading error as org.apache.spark.sql.execution.streaming.state.StateSchemaNotCompatible if restarting query in the same checkpoint location and changing their stateful operator. We need to catches such errors and throws a new error with informative message.
After physical planning, before execution phase, we will read from state metadata with the current operator id to fetch operator name of committed batch with the same operator id. If operator name does not match, throws the error.