Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-17861 Store data source partitions in metastore and push partition pruning into metastore
  3. SPARK-18185

Should fix INSERT OVERWRITE TABLE of Datasource tables with dynamic partitions

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.1.0
    • SQL
    • None

    Description

      As of current 2.1, INSERT OVERWRITE with dynamic partitions against a Datasource table will overwrite the entire table instead of only the updated partitions as in Hive. It also doesn't respect custom partition locations.

      We should delete only the proper partitions, scan the metastore for affected partitions with custom locations, and ensure that deletes/writes go to the right locations for those as well.

      Attachments

        Issue Links

          Activity

            People

              ekhliang Eric Liang
              ekhliang Eric Liang
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: