Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-21746

ArrayIndexOutOfBoundsException during dynamically partitioned hash join, with CBO disabled

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 4.0.0-alpha-1
    • Query Planning
    • None

    Description

      ArrayIndexOutOfBounds exception during query execution with dynamically partitioned hash join.
      Found on Hive 2.x. Seems to occur with CBO disabled/failed.
      Disabling constant propagation seems to allow the query to succeed.

      java.lang.ArrayIndexOutOfBoundsException: 203
              at org.apache.hadoop.hive.serde2.io.TimestampWritable.getTotalLength(TimestampWritable.java:217) ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
              at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.checkObjectByteInfo(LazyBinaryUtils.java:205) ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
              at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.parse(LazyBinaryStruct.java:142) ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
              at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.getFieldsAsList(LazyBinaryStruct.java:281) ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
              at org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$ReusableRowContainer.unpack(MapJoinBytesTableContainer.java:744) ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
              at org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$ReusableRowContainer.next(MapJoinBytesTableContainer.java:730) ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
              at org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$ReusableRowContainer.next(MapJoinBytesTableContainer.java:605) ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
              at org.apache.hadoop.hive.ql.exec.persistence.UnwrapRowContainer.next(UnwrapRowContainer.java:70) ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
              at org.apache.hadoop.hive.ql.exec.persistence.UnwrapRowContainer.next(UnwrapRowContainer.java:34) ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
              at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genUniqueJoinObject(CommonJoinOperator.java:819) ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
              at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:924) ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
              at org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:456) ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
              at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:359) ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
              at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:290) ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
              at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:319) ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
              at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:189) ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
              at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:172) ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
              at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:377) ~[tez-runtime-internals-0.8.4.2.6.4.119-3.jar:0.8.4.2.6.4.119-3]
              at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73) ~[tez-runtime-internals-0.8.4.2.6.4.119-3.jar:0.8.4.2.6.4.119-3]
              at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61) ~[tez-runtime-internals-0.8.4.2.6.4.119-3.jar:0.8.4.2.6.4.119-3]
              at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_112]
              at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_112]
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869) ~[hadoop-common-2.7.3.2.6.4.119-3.jar:?]
              at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61) ~[tez-runtime-internals-0.8.4.2.6.4.119-3.jar:0.8.4.2.6.4.119-3]
              at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37) ~[tez-runtime-internals-0.8.4.2.6.4.119-3.jar:0.8.4.2.6.4.119-3]
              at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) ~[tez-common-0.8.4.2.6.4.119-3.jar:0.8.4.2.6.4.119-3]
              at org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:118) ~[hive-llap-server-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
              at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_112]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_112]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_112]
              at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
      

      Attachments

        1. HIVE-21746.patch
          8 kB
          jcamachorodriguez
        2. HIVE-21746.patch
          8 kB
          jcamachorodriguez
        3. HIVE-21746.patch
          8 kB
          jcamachorodriguez
        4. HIVE-21746.hive2.patch
          7 kB
          Jason Dere

        Activity

          People

            jdere Jason Dere
            jdere Jason Dere
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: