Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-30034 Use Apache Hive 2.3 dependency by default
  3. SPARK-25193

insert overwrite doesn't throw exception when drop old data fails

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 1.6.3, 2.0.2, 2.1.3, 2.2.3, 2.3.4, 2.4.5
    • None
    • SQL

    Description

      dataframe.write.mode(SaveMode.Overwrite).insertInto(s"$databaseName.$tableName")

      Insert overwrite mode will drop old data in hive table if there's old data.

      But if data deleting fails, no exception will be thrown and the data folder will be like:

      hdfs://uxs_nbp/nba_score/dt=2018-08-15/seq_num=2/part-00000

      hdfs://uxs_nbp/nba_score/dt=2018-08-15/seq_num=2/part-000001534916642513.

      Two copies of data will be kept.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              xchen12138 chen xiao
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: