Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-2552

Runtime filter forwarding between operators

Agile BoardAttach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      Although the planner does much to reduce the output cardinality of operators where possible (e.g. by predicate pushdown), there are some situations where the planner cannot apply a selectivity-based optimisation, but the execution engine can based on the output from operators at runtime. One example is 'dynamic partition pruning', where a filter is computed on a join key which is a partition column by the build side of a hash table, and then propagated to the probe side scan. The scan can then simply skip those partitions which don't appear in the filters.

      There are other opportunities for optimisation based on runtime-computed filters. This JIRA tracks adding a general mechanism for filter computation and propagation in the backend, and the corresponding work in the planner to identify valid optimisation opportunities.

      Attachments

        Issue Links

        There are no Sub-Tasks for this issue.

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            henryr Henry Robinson
            henryr Henry Robinson
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment