Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-28302

SparkLauncher: The process cannot access the file because it is being used by another process

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.4.0
    • 2.3.4, 2.4.4, 3.0.0
    • Spark Core
    • None
    • Windows 7

    • Patch

    Description

      When using SparkLauncher to submit applications concurrently with a thread pool under Windows, some apps would show that "The process cannot access the file because it is being used by another process" and remains in LOST state at the end. (Issue can be reproduced with attach file.)

       

      After digging into the code, I find that, Windows cmd %RANDOM% would return the same number if we call it  instantly( < 500ms) after last call. As a result, SparkLauncher would get same output file(spark-class-launcher-output-%RANDOM%.txt) for apps. Then, the following app would hit the issue when it try to write the same file which has already been occupied by another app.

       

       

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Ngone51 wuyi
            Ngone51 wuyi
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment