Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-2087

Use separate thread pools for AsyncStreamTaskAdapter and AsyncRunLoop

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • container
    • None

    Description

      Currently, AsyncStreamTaskAdapter and AsyncRunLoop share thread pools and the thread pool size is governed by job.container.thread.pool.size. This introduces disparity in the semantics of task.max.concurrency between async stream task vs adapted async stream task.

      task.max.concurrency governs the parallelism within a task. In case of applications that directly implement AsyncStreamTask, this corresponds to maximum number of inflight messages for a task. However, in the case of applications that get adapted to AsyncStreamTask, the maximum number of inflight messages are bounded by the job.container.thread.pool.size. In fact, all the tasks within a container share the thread pool which results in even lesser parallelism within a task even though applications configure task.max.concurrency to a much greater number than job.container.thread.pool.size

       

       

       

      Attachments

        Issue Links

          Activity

            People

              bharathkk Bharath Kumarasubramanian
              bharathkk Bharath Kumarasubramanian
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 0.5h
                  0.5h