Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-2793 Make Pig Work on Windows without Cygwin
  3. PIG-2796

Local temporary paths are not always valid HDFS path names.

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.10.0
    • 0.11, 0.10.1, 0.12.0
    • None
    • None
    • Reviewed

    Description

      A number of pig scripts follow the pattern:

      File tempFile = File.createTempFile("this", ".txt");
      copyFromLocalToCluster (tempFile.to_string(), tempFile.to_string());
      tempFile.delete();

      The goal, here, seems to be to generate a temp filename to avoid issues on the next run if the file doesn't get cleaned up. The problem is that File.createTempFile on Windows creates files with names like C:\users\myuser\App data\local\temp\file.txt. The problem is that ":" is not a valid DFS character and so the put fails.

      The easy fix on this is to remove colons from the path before upload. Then we get something like C\users\myuser\App data\local\temp\file.txt which is a valid DFS pathname with minimal impact to the tests.

      Attachments

        Activity

          People

            jgordon John Gordon
            jgordon John Gordon
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: