Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
3.5.1
-
None
Description
In case of some flavours of nested joins involving repetition of relation, the projected columns when passed to the DataFrame.select API , as form of df.column , can result in plan resolution failure due to attribute resolution not happening.
A scenario in which this happens is
Project ( dataframe A.column("col-a") ) | Join2 | | Join1 DataFrame A | DataFrame A DataFrame B
In such cases, If it so happens that Join2 - right leg DataFrame A gets re-aliased due to De-Duplication of relations, and if the project uses Column definition obtained from DataFrame A, its exprId will not match the re-aliased Join2 - right Leg- DataFrame A , causing resolution failure.
Attachments
Issue Links
- Is contained by
-
SPARK-47320 Datasets involving self joins behave in an inconsistent and unintuitive manner
- Open
- links to