Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31990

Streaming's state store compatibility is broken

    XMLWordPrintableJSON

Details

    Description

      This line of https://github.com/apache/spark/pull/28062 changed the order of groupCols in dropDuplicates(). Thus, the executor JVM could probably crash, throw a random exception or even return a wrong answer when using the checkpoint written by the previous version. 

      Attachments

        Issue Links

          blocks

          Bug - A problem which impairs or prevents the functions of the product. SPARK-31894 Introduce UnsafeRow format validation for streaming state store

          • Major - Major loss of function.
          • Resolved
          incorporates

          Bug - A problem which impairs or prevents the functions of the product. SPARK-31905 Add compatibility tests for streaming state store format

          • Major - Major loss of function.
          • Resolved
          is caused by

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-31292 Replace toSet.toSeq with distinct for readability

          • Trivial - Cosmetic problem like misspelt words or misaligned text.
          • Resolved
          relates to

          New Feature - A new feature of the product, which has yet to be developed. SPARK-19497 dropDuplicates with watermark

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          links to

          Pull request #28830 [Github] Pull Request #28830 (maropu)

          Pull request #28830 [Github] Pull Request #28830 (maropu)

          Activity

            People

              maropu Takeshi Yamamuro
              smilegator Xiao Li
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: