Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-2552

Runtime filter forwarding between operators

    XMLWordPrintableJSON

Details

    Description

      Although the planner does much to reduce the output cardinality of operators where possible (e.g. by predicate pushdown), there are some situations where the planner cannot apply a selectivity-based optimisation, but the execution engine can based on the output from operators at runtime. One example is 'dynamic partition pruning', where a filter is computed on a join key which is a partition column by the build side of a hash table, and then propagated to the probe side scan. The scan can then simply skip those partitions which don't appear in the filters.

      There are other opportunities for optimisation based on runtime-computed filters. This JIRA tracks adding a general mechanism for filter computation and propagation in the backend, and the corresponding work in the planner to identify valid optimisation opportunities.

      Attachments

        Issue Links

          Activity

            People

              henryr Henry Robinson
              henryr Henry Robinson
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: