Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-11339

Cache clean up might fail on Windows

Details

    • Improvement
    • Status: Open
    • P3
    • Resolution: Unresolved
    • None
    • None
    • runner-py-interactive
    • None

    Description

      Details can be found in this PR:
      https://github.com/apache/beam/pull/12779

      apache_beam\runners\interactive\recording_manager_test.py:75: 
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
      apache_beam\runners\interactive\interactive_environment.py:118: in new_env
          _interactive_beam_env.cleanup()
      apache_beam\runners\interactive\interactive_environment.py:272: in cleanup
          cache_manager.cleanup()
      apache_beam\runners\interactive\caching\streaming_cache.py:391: in cleanup
          shutil.rmtree(self._cache_dir)
      c:\hostedtoolcache\windows\python\3.6.8\x64\lib\shutil.py:500: in rmtree
          return _rmtree_unsafe(path, onerror)
      c:\hostedtoolcache\windows\python\3.6.8\x64\lib\shutil.py:390: in _rmtree_unsafe
          _rmtree_unsafe(fullname, onerror)
      c:\hostedtoolcache\windows\python\3.6.8\x64\lib\shutil.py:395: in _rmtree_unsafe
          onerror(os.unlink, fullname, sys.exc_info())
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
      
      ...
      
      >                   os.unlink(fullname)
      E                   PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'D:\\a\\beam\\beam\\sdks\\python\\target\\.tox\\py36-win\\tmp\\it-4m1c1oje2145793178144\\full\\fb91a47796-2145832985040-2145832986608-2145793178144'
      

      This does not happen on Linux-like systems:
      https://user-images.githubusercontent.com/4423149/100034273-b343ef80-2db0-11eb-8988-dcfc2c79322c.png

      A potential solution is to enable ignore_errors and log about manually cleaning up onerror.

      Attachments

        Activity

          People

            Unassigned Unassigned
            ningk Ning
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: