Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-3825

Distribute runtime filter aggregation across cluster

    XMLWordPrintableJSON

Details

    Description

      Runtime filters can be tens of MB or more, and incasting all filters from all shuffle joins to the coordinator can put a lot of memory pressure on that node. To alleviate this we should consider spreading out the aggregation operation across the cluster, so that a different node aggregates each runtime filter.

      This still restricts aggregation to #runtime-filters nodes, which will usually be less than the cluster size. If we want to smooth that out further we could use tree-based aggregation, but let's measure the benefits of simply distributing the aggregation work first.

      Attachments

        Issue Links

          causes

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-13040 SIGSEGV in QueryState::UpdateFilterFromRemote

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          is depended upon by

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-6412 Memory issues with processing of incoming global runtime filters on coordinator

          • Major - Major loss of function.
          • Resolved
          is related to

          Sub-task - The sub-task of the issue IMPALA-8687 --rpc_use_loopback may not work for runtime filter RPCs

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-7486 Admit less memory on dedicated coordinator for admission control purposes

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-4457 Coordinator sends out runtime filters serially

          • Major - Major loss of function.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-6144 Coordinator threads that publish RuntimeFilters continue to run after query failure/cancellation

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. IMPALA-3701 Evaluate compressing Runtime filters to save coordinator network bandwidth

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              rizaon Riza Suminto
              henryr Henry Robinson
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: