Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-8950

Add -d and -f option to copyFromLocal and re-enable disabled S3 tests

    XMLWordPrintableJSON

Details

    • Test
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • Impala 3.4.0
    • None
    • None
    • ghx-label-7

    Description

      The -d option for hdfs dfs -copyFromLocal "Skip[s] creation of temporary file with the suffix .COPYING". The -f option "Overwrites the destination if it already exists".

      By using the -d option, copies to S3 avoid the additional overhead of copying data to a tmp file and then renaming the file. The -f option overwrites the file if it exists, which should be safe since tests should be writing to unique directories anyway. With HADOOP-16490, create(overwrite=true) avoids issuing a HEAD request on the path, which prevents any cached 404s on the S3 key.

      After these changes, the tests disabled by IMPALA-8189 can be re-enabled.

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-8581 Switch hdfs commandline calls in dataload and tests to use "-d" on s3 where applicable

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-8189 TestParquet.test_resolution_by_name fails on S3 because 'hadoop fs -cp' fails

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-8344 Add support for running S3 tests with S3Guard

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          relates to

          Test - A new unit, integration or system test. IMPALA-9004 TestCompressedFormats is broken for text files

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              stakiar Sahil Takiar
              stakiar Sahil Takiar
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: