Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-17232

"No match found" Compactor finds a bucket file thinking it's a directory

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • None
    • Transactions
    • None

    Description

      2017-08-02T12:38:11,996  WARN [main] compactor.CompactorMR: Found a non-bucket file that we thought matched the bucket pattern! file:/Users/ekoifman/dev/hiv\
      erwgit/ql/target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands2-1501702264311/warehouse/acidtblpart/p=1/delta_0000013_0000013_0000/bucket_00001 Matcher=java\
      .util.regex.Matcher[pattern=^[0-9]{6} region=0,12 lastmatch=]
      2017-08-02T12:38:11,996  INFO [main] mapreduce.JobSubmitter: Cleaning up the staging area file:/tmp/hadoop/mapred/staging/ekoifman1723152463/.staging/job_lo\
      cal1723152463_0183
      2017-08-02T12:38:11,997 ERROR [main] compactor.Worker: Caught exception while trying to compact id:1,dbname:default,tableName:ACIDTBLPART,partName:null,stat\
      e:^@,type:MAJOR,properties:null,runAs:null,tooManyAborts:false,highestTxnId:0.  Marking failed to avoid repeated failures, java.lang.IllegalStateException: \
      No match found
              at java.util.regex.Matcher.group(Matcher.java:536)
              at java.util.regex.Matcher.group(Matcher.java:496)
              at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorInputFormat.addFileToMap(CompactorMR.java:577)
              at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorInputFormat.getSplits(CompactorMR.java:549)
              at org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:330)
              at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:322)
              at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:198)
              at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
              at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1338)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:422)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
              at org.apache.hadoop.mapreduce.Job.submit(Job.java:1338)
              at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:575)
              at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:570)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:422)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
              at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:570)
              at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:561)
              at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.launchCompactionJob(CompactorMR.java:320)
              at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.run(CompactorMR.java:275)
              at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:166)
              at org.apache.hadoop.hive.ql.TestTxnCommands2.runWorker(TestTxnCommands2.java:1138)
              at org.apache.hadoop.hive.ql.TestTxnCommands2.updateDeletePartitioned(TestTxnCommands2.java:894)
      

      the stack trace points to 1st runWorker() in updateDeletePartitioned() though the test run was TestTxnCommands2WithSplitUpdateAndVectorization

      Attachments

        1. HIVE-17232.01.patch
          3 kB
          Steve Yeom

        Issue Links

          Activity

            People

              ekoifman Eugene Koifman
              ekoifman Eugene Koifman
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: