Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-25318 Improvement of scheduler and execution for Flink OLAP
  3. FLINK-26675

Parallelized heavy serialization operations in StreamingJobGraphGenerator

    XMLWordPrintableJSON

Details

    Description

      For some complex jobs, the compilation time of the job graph may exceed one second, which is unaffordable for short queries in the OLAP scenario. After a deeper investigation, we found that the hot path is the serialization of `OperatorCoordinator` and `StreamOperatorFactory`.

      We proposed to parallelized these heavy serialization operations to accelerate the compilation of job graphs.

      Attachments

        Issue Links

          causes

          Bug - A problem which impairs or prevents the functions of the product. FLINK-29576 Serialized OperatorCoordinators are collected in a not thread-safe ArrayList even though serialization was parallelized in FLINK-26675

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. FLINK-29431 Exceptions during job graph serialization lock up client

          • Blocker - Blocks development and/or testing work, production could not run
          • Closed
          links to

          Web Link GitHub Pull Request #19108

          Activity

            People

              guoyangze Yangze Guo
              guoyangze Yangze Guo
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: