Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31459

When using the insert overwrite directory syntax, if the target path is an existing file, the final run result is incorrect

    XMLWordPrintableJSON

Details

    • Bug
    • Status: In Progress
    • Major
    • Resolution: Unresolved
    • 2.4.5, 3.0.0
    • None
    • SQL
    • spark2.4.5

    • Patch

    Description

      When using the insert overwrite directory syntax, if the target path is an existing file, the final operation result is incorrect.
      At present, Spark will not delete the existing files. After the calculation is completed, one of the result files will be renamed to the result path.
      This is different from hive's behavior. Hive will delete the existing target file.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              mcdull_zhang mcdull_zhang
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated: