Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-10199

Separate state restoration into separate threads

    XMLWordPrintableJSON

Details

    Description

      As part of the restoration optimization effort, we would like to move the restoration process to separate threads such that:

      1. Stream threads would not be restricted by the main consumer `poll` frequency to keep as part of the group.

      2. We can allow larger batches of data to be written into the restoration.

      Besides this, we'd also like to fix the known issues that for piggy-backed source topics as changelog topics, the serde exception / extra processing logic would be skipped.

      We would also cleanup the global update tasks as part of this effort to consolidate to the separate restoration threads, and would also gear them up with corresponding monitoring metrics (KIPs in progress).

      Attachments

        Issue Links

          blocks

          Improvement - An improvement or enhancement to an existing feature or task. KAFKA-14192 Move registering and unregistering changelogs to state updater

          • Major - Major loss of function.
          • Open
          contains

          Bug - A problem which impairs or prevents the functions of the product. KAFKA-13295 Long restoration times for new tasks can lead to transaction timeouts

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. KAFKA-10575 StateRestoreListener#onRestoreEnd should always be triggered

          • Major - Major loss of function.
          • Resolved
          is blocked by

          Sub-task - The sub-task of the issue KAFKA-16336 Remove Deprecated metric standby-process-ratio

          • Blocker - Blocks development and/or testing work, production could not run
          • Open

          Task - A task that needs to be done. KAFKA-16567 Add New Stream Metrics based on KIP-869

          • Major - Major loss of function.
          • Open
          is related to

          Bug - A problem which impairs or prevents the functions of the product. KAFKA-14548 Stable streams applications stall due to infrequent restoreConsumer polls

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. KAFKA-7934 Optimize restore for windowed and session stores

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. KAFKA-8023 Improve global state store restoration by using multiple update threads

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. KAFKA-13239 Use RocksDB.ingestExternalFile for restoration

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. KAFKA-8037 KTable restore may load bad data

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. KAFKA-13500 Consider adding a dedicated standby consumer

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. KAFKA-7663 Custom Processor supplied on addGlobalStore is not used when restoring state from topic

          • Major - Major loss of function.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. KAFKA-10005 Decouple RestoreListener from RestoreCallback and not enable bulk loading for RocksDB

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #8988

          Web Link GitHub Pull Request #11499

          Web Link GitHub Pull Request #12128

          Web Link GitHub Pull Request #12200

          Web Link GitHub Pull Request #12270

          Web Link GitHub Pull Request #12279

          Web Link GitHub Pull Request #12312

          Web Link GitHub Pull Request #12337

          Web Link GitHub Pull Request #12338

          Web Link GitHub Pull Request #12379

          Web Link GitHub Pull Request #12384

          Web Link GitHub Pull Request #12386

          Web Link GitHub Pull Request #12387

          Web Link GitHub Pull Request #12391

          Web Link GitHub Pull Request #12397

          Web Link GitHub Pull Request #12427

          Web Link GitHub Pull Request #12439

          Web Link GitHub Pull Request #12442

          Web Link GitHub Pull Request #12466

          Web Link GitHub Pull Request #12497

          Web Link GitHub Pull Request #12519

          Web Link GitHub Pull Request #12520

          Web Link GitHub Pull Request #12521

          Web Link GitHub Pull Request #12532

          Web Link GitHub Pull Request #12547

          Web Link GitHub Pull Request #12549

          Web Link GitHub Pull Request #12554

          Web Link GitHub Pull Request #12562

          Web Link GitHub Pull Request #12569

          Web Link GitHub Pull Request #12573

          Web Link GitHub Pull Request #12583

          Web Link GitHub Pull Request #12600

          Web Link GitHub Pull Request #12638

          Web Link GitHub Pull Request #12650

          Web Link GitHub Pull Request #12659

          Web Link GitHub Pull Request #12687

          Web Link GitHub Pull Request #13228

          Web Link GitHub Pull Request #13300

          Web Link GitHub Pull Request #13829

          Web Link GitHub Pull Request #13925

          Web Link GitHub Pull Request #13927

          Web Link GitHub Pull Request #14145

          Web Link GitHub Pull Request #14214

          Web Link GitHub Pull Request #14249

          Web Link GitHub Pull Request #14265

          Web Link GitHub Pull Request #14403

          Web Link GitHub Pull Request #14437

          Web Link GitHub Pull Request #14442

          Web Link GitHub Pull Request #15849

          Web Link GitHub Pull Request #15852

          Web Link GitHub Pull Request #15867

          Web Link GitHub Pull Request #15870

          Web Link GitHub Pull Request #15871

          Web Link GitHub Pull Request #15882

          Web Link GitHub Pull Request #15894

          Web Link GitHub Pull Request #15896

          mentioned in

          Page Loading...

          Activity

            People

              cadonna Bruno Cadonna
              guozhang Guozhang Wang
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated: