Structured Streaming - Arbitrary State API v2

We are planning to introduce a new operator for Spark Structured Streaming around arbitrary stateful operations. The operator(s) we have today (mapGroupsWithState/flatMapGroupsWithState) have a bunch of limitations (as noted in Q3 below) that prevent the use of the APIs for complex/advanced use-cases for stateful stream processing. We plan to remove these limitations with the introduction of a new API to handle state manipulation for streaming queries.

As part of this effort, we want to address:

• composite types

• multiple state variables (flexible data modeling)

• state TTL

• callback/timer improvements

• schema evolution

• initial state

• output modes

• co-process functions

• chaining operators after new operator etc