Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-40025

Project Lightspeed: Faster and Simpler Stream Processing with Apache Spark

    XMLWordPrintableJSON

Details

    • Umbrella
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.2.2
    • None
    • Structured Streaming
    • None

    Description

      Project Lightspeed is an umbrella project aimed at improving a couple of key aspects of Spark Streaming:

      • Improving the latency and ensuring it is predictable
      • Enhancing functionality for processing data with new operators and APIs

       

      Umbrella Jira to track all tickets under Project Lightspeed

      SPARK-39585 - Multiple Stateful Operators in Structured Streaming
      SPARK-39586 - Advanced Windowing in Structured Streaming
      SPARK-39587 - Schema Evolution for Stateful Pipelines
      SPARK-39589 - Asynchronous I/O support
      SPARK-40431 - Python API for Arbitrary Stateful Processing
      SPARK-39591 - Offset Management Improvements

      SPARK-40849 - Async log purge

      SPARK-39592 - Asynchronous State Checkpointing
      SPARK-39593 - Configurable State Checkpointing Frequency

      Attachments

        Issue Links

          contains

          New Feature - A new feature of the product, which has yet to be developed. SPARK-39586 Advanced Windowing in Structured Streaming

          • Major - Major loss of function.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. SPARK-39589 Asynchronous I/O API support in Structured Streaming

          • Major - Major loss of function.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. SPARK-40849 Async log purge

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. SPARK-40957 Add in memory cache in HDFSMetadataLog

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-39585 Multiple Stateful Operators in Structured Streaming

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-39587 Schema Evolution for Stateful Pipelines in Structured Streaming

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-39592 Asynchronous State Checkpointing in Structured Streaming

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-39593 Configurable State Checkpointing Frequency in Structured Streaming

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-39590 Python API Parity in Structure Streaming

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-39591 SPIP: Asynchronous Offset Management in Structured Streaming

          • Major - Major loss of function.
          • Resolved
          Is contained by

          New Feature - A new feature of the product, which has yet to be developed. SPARK-40957 Add in memory cache in HDFSMetadataLog

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              jerrypeng Boyang Jerry Peng
              Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated: