Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-15142

CompactorMR fails with FileNotFoundException

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Not A Problem
    • 1.0.0
    • None
    • Transactions
    • None

    Description

      No of maps and reduces are 0 job_1478131263487_0009
Job commit failed: java.io.FileNotFoundException: File hdfs://....../_tmp_80e50691-c9fe-485a-8e1e-ba20331c7d97 does not exist.
at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:904)
at org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:113)
at org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:966)
at org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:962)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:962)
at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorOutputCommitter.commitJob(CompactorMR.java:776)
at org.apache.hadoop.mapred.OutputCommitter.commitJob(OutputCommitter.java:291)
at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:285)
at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:237)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

      when Compactor doesn't have permissions to write to the table dir.

      Evidently we loose the exception when creating CompactorMR.TMP_LOCATION and only see that the file is not there on commit of the job

      Attachments

        Issue Links

          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. HIVE-15362 Add the missing fields for 2.2.0 upgrade scripts

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. HIVE-18044 CompactorMR.CompactorOutputCommitter.abortTask() not implemented

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              wzheng Wei Zheng
              ekoifman Eugene Koifman
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: