Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-19779

structured streaming exist needless tmp file

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 2.0.3, 2.1.1, 2.2.0
    • 2.0.3, 2.1.1, 2.2.0
    • Structured Streaming
    • None

    Description

      The PR (https://github.com/apache/spark/pull/17012) can to fix restart a Structured Streaming application using hdfs as fileSystem, but also exist a problem that a tmp file of delta file is still reserved in hdfs. And Structured Streaming don't delete the tmp file generated when restart streaming job in future, so we need to delete the tmp file after restart streaming job.

      Attachments

        Issue Links

          Activity

            People

              guifengleaf@gmail.com Feng Gui
              guifengleaf@gmail.com Feng Gui
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: