Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-11132

Front-end test PlannerTest.testResourceRequirements can fail

    XMLWordPrintableJSON

Details

    • Test
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • None
    • None
    • None

    Description

      The test miscalculates per-host memory requirements, apparently due to an incorrect HBase cardinality estimate:

      Section DISTRIBUTEDPLAN of query:
      select * from functional_hbase.alltypessmall
      
      Actual does not match expected result:
      Max Per-Host Resource Reservation: Memory=4.00MB Threads=2
      Per-Host Resource Estimates: Memory=10MB
      Codegen disabled by planner
      Analyzed query: SELECT * FROM functional_hbase.alltypessmall
      
      F01:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
      |  Per-Host Resources: mem-estimate=5.08MB mem-reservation=4.00MB thread-reservation=1
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      PLAN-ROOT SINK
      |  output exprs: functional_hbase.alltypessmall.id, functional_hbase.alltypessmall.bigint_col, functional_hbase.alltypessmall.bool_col, functional_hbase.alltypessmall.date_string_col, functional_hbase.alltypessmall.double_col, functional_hbase.alltypessmall.float_col, functional_hbase.alltypessmall.int_col, functional_hbase.alltypessmall.month, functional_hbase.alltypessmall.smallint_col, functional_hbase.alltypessmall.string_col, functional_hbase.alltypessmall.timestamp_col, functional_hbase.alltypessmall.tinyint_col, functional_hbase.alltypessmall.year
      |  mem-estimate=4.00MB mem-reservation=4.00MB spill-buffer=2.00MB thread-reservation=0
      |
      01:EXCHANGE [UNPARTITIONED]
      |  mem-estimate=1.08MB mem-reservation=0B thread-reservation=0
      |  tuple-ids=0 row-size=89B cardinality=28.57K
      |  in pipelines: 00(GETNEXT)
      |
      F00:PLAN FRAGMENT [RANDOM] hosts=3 instances=3
      Per-Host Resources: mem-estimate=4.00KB mem-reservation=0B thread-reservation=1
      00:SCAN HBASE [functional_hbase.alltypessmall]
         stored statistics:
           table: rows=100
           columns: all
         mem-estimate=4.00KB mem-reservation=0B thread-reservation=0
         tuple-ids=0 row-size=89B cardinality=28.57K
         in pipelines: 00(GETNEXT)
      
      Expected:
      Max Per-Host Resource Reservation: Memory=4.00MB Threads=2
      Per-Host Resource Estimates: Memory=10MB
      Codegen disabled by planner
      Analyzed query: SELECT * FROM functional_hbase.alltypessmall
      
      F01:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
      |  Per-Host Resources: mem-estimate=4.02MB mem-reservation=4.00MB thread-reservation=1
      PLAN-ROOT SINK
      |  output exprs: functional_hbase.alltypessmall.id, functional_hbase.alltypessmall.bigint_col, functional_hbase.alltypessmall.bool_col, functional_hbase.alltypessmall.date_string_col, functional_hbase.alltypessmall.double_col, functional_hbase.alltypessmall.float_col, functional_hbase.alltypessmall.int_col, functional_hbase.alltypessmall.month, functional_hbase.alltypessmall.smallint_col, functional_hbase.alltypessmall.string_col, functional_hbase.alltypessmall.timestamp_col, functional_hbase.alltypessmall.tinyint_col, functional_hbase.alltypessmall.year
      |  mem-estimate=4.00MB mem-reservation=4.00MB spill-buffer=2.00MB thread-reservation=0
      |
      01:EXCHANGE [UNPARTITIONED]
      |  mem-estimate=16.00KB mem-reservation=0B thread-reservation=0
      |  tuple-ids=0 row-size=89B cardinality=50
      |  in pipelines: 00(GETNEXT)
      |
      F00:PLAN FRAGMENT [RANDOM] hosts=3 instances=3
      Per-Host Resources: mem-estimate=4.00KB mem-reservation=0B thread-reservation=1
      00:SCAN HBASE [functional_hbase.alltypessmall]
         stored statistics:
           table: rows=100
           columns: all
         mem-estimate=4.00KB mem-reservation=0B thread-reservation=0
         tuple-ids=0 row-size=89B cardinality=50
         in pipelines: 00(GETNEXT)
      

      Attachments

        Activity

          People

            sql_forever Qifan Chen
            sql_forever Qifan Chen
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: