Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-47331

Serialization using case classes/primitives/POJO based on SQL encoder for Arbitrary State API v2.

    XMLWordPrintableJSON

Details

    Description

      In the new operator for arbitrary state-v2, we cannot rely on the session/encoder being available since the initialization for the various state instances happens on the executors. Also, we can only support limited state types with the available encoders. Hence, for the state serialization, we propose to serialize primitives/case classes/POJO with SQL encoder. Leveraging SQL encoder can speed up the serialization.

      Attachments

        Issue Links

          Activity

            People

              jing.zhan Jing Zhan
              jing.zhan Jing Zhan
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: