Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-18593

NPE on vectorization group by

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 2.3.2
    • None
    • Hive, HiveServer2
    • None

    Description

      Vectorisation with some queries seem to be failing with null pointer exceptions. This happens only with 2.3.2 release and not the older ones.

      In the case, (in BytesColumnVector.java, vector[0] is null, isRepeating is true, length[0] is 0, start[0] is 0

      public void copySelected(
          boolean selectedInUse, int[] sel, int size, BytesColumnVector output) {
      
        // Output has nulls if and only if input has nulls.
        output.noNulls = noNulls;
        output.isRepeating = false;
      
        // Handle repeating case
        if (isRepeating) {
          output.setVal(0, vector[0], start[0], length[0]);
          output.isNull[0] = isNull[0];
          output.isRepeating = true;
          return;
        }
      

      Exception trace below

      Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row
      at org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:883)
      at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:86)
      ... 17 more
      Caused by: java.lang.NullPointerException
      at java.lang.System.arraycopy(Native Method)
      at org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173)
      at org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:321)
      at org.apache.hadoop.hive.ql.exec.vector.expressions.IfExprStringGroupColumnStringGroupColumn.evaluate(IfExprStringGroupColumnStringGroupColumn.java:85)
      at org.apache.hadoop.hive.ql.exec.vector.expressions.aggregates.gen.VectorUDAFMaxString.aggregateInputSelection(VectorUDAFMaxString.java:135)
      at org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeBase.processAggregators(VectorGroupByOperator.java:218)
      at org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.doProcessBatch(VectorGroupByOperator.java:408)
      at org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeBase.processBatch(VectorGroupByOperator.java:179)
      at org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.process(VectorGroupByOperator.java:1021)
      at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
      at org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:137)
      at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
      at org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.process(VectorFilterOperator.java:123)
      at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
      at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
      at org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:783)
      ... 18 more
      

       

      Table details 

      CREATE TABLE `test_table`(
       `a` string,
       `b` string,
       `c` string)
       ROW FORMAT SERDE
       'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
      STORED AS INPUTFORMAT
       'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
       OUTPUTFORMAT
       'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
       TBLPROPERTIES (
       'orc.compress'='SNAPPY',
       'orc.compress.size'='262144',
       'orc.create.index'='true',
       'orc.row.index.stride'='10000',
       'orc.stripe.size'='268435456',
       'transient_lastDdlTime'='1517556432');

      Query

      SELECT
       NVL(Max(CASE WHEN b IN ('some_literal') THEN b ELSE c END),'') AS scol
      FROM test_table
      GROUP BY a
      LIMIT 1000;

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              amrk7 Amruth Sampath
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated: