Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-47305

PruneFilters incorrectly tags isStreaming flag when replacing child of Filter with LocalRelation

    XMLWordPrintableJSON

Details

    Description

      This seems to be a very old bug in optimizer. Related ticket:  https://issues.apache.org/jira/browse/SPARK-21765

      When filter is evaluated to be always false, PruneFilters replaces the filter with empty LocalRelation, which effectively prunes filter. The logic cares about migration of the isStreaming flag, but incorrectly migrated in some case, via picking up the value of isStreaming flag from root node rather than filter (or child).

      isStreaming flag is true if the value of isStreaming flag from any of children is true. Flipping the coin, some children might have isStreaming flag as "false". If the filter being pruned is a descendant to such children (in other word, ancestor of streaming node), LocalRelation is incorrectly tagged as streaming where it should be batch.

      Attachments

        Activity

          People

            kabhwan Jungtaek Lim
            kabhwan Jungtaek Lim
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: