Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-8021

Hadoop ignores old-style config options for enabling compressed output when passed on from PIG

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 0.23.1
    • 0.23.1
    • conf
    • None

    Description

      Hadoop seems to ignore the config options even though they are printed as deprecation warnings in the log: mapred.output.compress and
      mapred.output.compression.codec

      • settings that work on 0.20 but not on 0.23
        mapred.output.compress=true
        mapred.output.compression.codec=org.apache.hadoop.io.compress.BZip2Codec
      • settings that work on 0.23
        mapreduce.output.fileoutputformat.compress=true
        mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.BZip2Codec

      This breaks backwards compatibility and causes existing jobs to fail.

      This is different from HADOOP-7993 in that that one fixes any jobs run from the CLI but PIG jobs are still facing this issue.

      Attachments

        Issue Links

          Activity

            People

              anupamseth Anupam Seth
              anupamseth Anupam Seth
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: