Description
Spark can't push down filter condition of Or:
Such as if I have a table default.test, his partition col is "dt",
if I use query :
select * from default.test where dt=20190625 or (dt = 20190626 and id in (1,2,3) )
In this case, Spark will resolve or condition as one expression, and since this expr has reference of "id", then it can't been push down.
Attachments
Issue Links
- causes
-
SPARK-32284 Avoid expanding too many CNF predicates in partition pruning
-
- In Progress
-
- is duplicated by
-
-
- Resolved
-
- links to
