Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-24780

DataFrame.column_name should resolve to a distinct ref

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • 3.1.0
    • None
    • PySpark, SQL
    • None

    Description

      If we join a dataframe with another dataframe which has the same column name of the conditions (e.g. shared lineage on one of the conditions) even though the join condition may be written with the full name, the columns returned don't have the dataframe alias and as such will create a cross-join.

      For example this currently works even if both posts_by_sampled_authors  &  mailing_list_posts_in_reply_to contain both in_reply_to and message_id fields.

       

      posts_with_replies = posts_by_sampled_authors.join(
       mailing_list_posts_in_reply_to,
       [F.col("mailing_list_posts_in_reply_to.in_reply_to") == F.col("posts_by_sampled_authors.message_id")],
       "inner")

       

      But a similarly written expression:

      posts_with_replies = posts_by_sampled_authors.join(
       mailing_list_posts_in_reply_to,
       [mailing_list_posts_in_reply_to.in_reply_to == posts_by_sampled_authors.message_id],
       "inner")

      will fail.

       

      I'm not super sure whats going on inside of the resolution that's causing it to get confused.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              holden Holden Karau
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated: