Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-12498

MERGING-EXCHANGE can consume high memory if number of sender is high

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • Impala 4.3.0
    • None
    • Distributed Exec
    • None
    • ghx-label-7

    Description

      MERGING-EXCHANGE dedicate one row batch queue per data stream sender with soft limit of 10MB per queue. For large cluster with hundreds of backend executor, this can lead MERGING-EXCHANGE to require hundreds of queues totaling hundreds MBs of memory.

      We need to improve the design to keep memory of coordinator fragment low.

      Attachments

        Activity

          People

            Unassigned Unassigned
            rizaon Riza Suminto
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: