Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-4234

Compressor can cause IllegalArgumentException in Buffer.limit where limit exceeds capacity

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • None
    • 0.10.0, 0.9.3
    • None

    Description

      java.lang.IllegalArgumentException
        at java.nio.Buffer.limit(Buffer.java:275)
        at org.apache.hadoop.io.compress.lz4.Lz4Compressor.compress(Lz4Compressor.java:240)
        at org.apache.hadoop.io.compress.BlockCompressorStream.compress(BlockCompressorStream.java:149)
        at org.apache.hadoop.io.compress.BlockCompressorStream.write(BlockCompressorStream.java:131)
        at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58)
        at java.io.DataOutputStream.write(DataOutputStream.java:107)
        at org.apache.tez.runtime.library.common.sort.impl.IFile$Writer.writeKVPair(IFile.java:423)
        at org.apache.tez.runtime.library.common.sort.impl.IFile$Writer.append(IFile.java:392)
        at org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter.spill(DefaultSorter.java:927)
        at org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter.sortAndSpill(DefaultSorter.java:865)
        at org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter.flush(DefaultSorter.java:729)
        at org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput.close(OrderedPartitionedKVOutput.java:191)
        at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.close(LogicalIOProcessorRuntimeTask.java:398)
        at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:83)
        at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:2035)
        at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
        at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
        at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
      

      Attachments

        1. TEZ-4234.wip.patch
          1 kB
          László Bodor
        2. TEZ-4234.repro.patch
          2 kB
          László Bodor
        3. TEZ-4234.01.patch
          27 kB
          László Bodor
        4. TEZ-4234.02.patch
          30 kB
          László Bodor
        5. TEZ-4234.03.patch
          31 kB
          László Bodor
        6. TEZ-4234.04.branch-0.9.patch
          30 kB
          László Bodor

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            abstractdog László Bodor
            jeagles Jonathan Turner Eagles
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment