Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25709

Can't append to a data source V2 unless target already has schema information or fixed schema matching dataframe

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 2.4.0
    • None
    • SQL
    • None

    Description

      Let's assume we have a writer data source which writes to a dynamic schema (e.g. writing to a database table). If we have mode append but the table is missing then writing will fail.

       

      The reason for this is that in the dataframeWriter save method we have val relation = DataSourceV2Relation.create(source, options) which in turn contains: val reader = source.createReader(options, userSpecifiedSchema)

      The problem is that this sets the schema from the reader portion of the data source. If the data source can't infer the schema then we will fail to write (even thought the schema is actually part of the dataframe).

       

       

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              assaf.mendelson Assaf Mendelson
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: