Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-17406

UDAF throws IllegalArgumentException for a complex input when column stats is not provided

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Duplicate
    • 2.3.0
    • None
    • Query Processor
    • None

    Description

      I found that UDAF (both generic and non-generic UDAF w/ or w/o estimable) of Hive v2.3.0 throws IllegalArgumentException for a complex input when column stats is not provided.

      The exception does not occur in v2.1.0.

      https://github.com/apache/hive/blob/34eebff194e81180202d198200e84058c4910d95/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java#L1156

      select version();
      > 2.3.0-amzn-0 rcb482944667f96f43c89932dcb66d61ee7e4ac1d
      
      with t2 as ( 
        select array(1,2) as c1 
        union all 
        select array(2,3) as c1
      ) 
      select collect_list(c1) from t2;
      
      > FAILED: IllegalArgumentException Size requested for unknown type: java.util.Collection
      

      On the other hand, it succeeds when colunm stats is provided as follows:

      create table t1 as (
        select array(1,2) as c1 
        union all
        select array(2,3) as c1
      );
      
      > select collect_list(c1) from t1;
      [[1,2],[2,3]]
      
      > desc formatted t1;
      ...       
      Table Parameters:                
              COLUMN_STATS_ACCURATE   {\"BASIC_STATS\":\"true\"}
              numFiles                2                   
              numRows                 2                   
              rawDataSize             6                   
              totalSize               8                   
              transient_lastDdlTime   1503990290
      

      Attachments

        1. HIVE-17406.1.patch
          4 kB
          Zoltan Haindrich

        Issue Links

          Activity

            People

              aserkm A
              myui Makoto Yui
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: