Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-21558

Query based compaction fails if the temporary FS is different than the table FS

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 4.0.0-alpha-1
    • Transactions
    • None

    Description

      The Exception I got is like this:

      2019-04-01T13:45:44,035 ERROR [PeterVary-MBP15.local-33] compactor.Worker: Caught exception while trying to compact id:24,dbname:default,tableName:acid,partName:null,state:,type:MAJOR,properties:null,runAs:petervary,tooManyAborts:false,highestWriteId:9. Marking failed to avoid repeated failures, java.lang.IllegalArgumentException: Wrong FS: pfile:/Users/petervary/data/apache/hive/warehouse/acid/base_0000009_v0000284/bucket_00000, expected: file:///
      at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:781)
      at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:86)
      at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:636)
      at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:930)
      at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:631)
      at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:454)
      at org.apache.hadoop.fs.FileSystem.isFile(FileSystem.java:1768)
      at org.apache.hadoop.hive.ql.io.ProxyLocalFileSystem.rename(ProxyLocalFileSystem.java:34)
      at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.commitCrudMajorCompaction(CompactorMR.java:583)
      at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.runCrudCompaction(CompactorMR.java:401)
      at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.run(CompactorMR.java:248)
      at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:195)

      Attachments

        1. HIVE-21558.patch
          2 kB
          Peter Vary
        2. HIVE-21558.02.patch
          2 kB
          Peter Vary

        Activity

          People

            pvary Peter Vary
            pvary Peter Vary
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: