Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-3087

Explain plan regression due to codegen of disjunct predicates

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • Impala 2.5.0
    • Impala 2.5.0
    • Frontend

    Description

      TPC-DS Q3 has regressed in 2.5 compared to 2.3 due to query planning taking 6s compared to 1.3s for 2.3

      Query

      select
        dt.d_year,
        item.i_brand_id brand_id,
        item.i_brand brand,
        sum(ss_ext_sales_price) sum_agg
      from
        date_dim dt,
        store_sales,
        item
      where
        dt.d_date_sk = store_sales.ss_sold_date_sk
        and store_sales.ss_item_sk = item.i_item_sk
        and item.i_manufact_id = 436
        and dt.d_moy = 12
        -- partition key filters
        and (ss_sold_date_sk between 2451149 and 2451179
          or ss_sold_date_sk between 2451514 and 2451544
          or ss_sold_date_sk between 2451880 and 2451910
          or ss_sold_date_sk between 2452245 and 2452275
          or ss_sold_date_sk between 2452610 and 2452640)
      group by
        dt.d_year,
        item.i_brand,
        item.i_brand_id
      order by
        dt.d_year,
        sum_agg desc,
        brand_id
      limit 100
      

      From the profile

          Planner Timeline: 6s103ms
             - Analysis finished: 10.923ms (10.923ms)
             - Equivalence classes computed: 11.240ms (316.228us)
             - Single node plan created: 6s095ms (6s083ms)
             - Runtime filters computed: 6s095ms (142.439us)
             - Distributed plan created: 6s098ms (3.806ms)
             - Lineage info computed: 6s100ms (1.060ms)
             - Planning finished: 6s103ms (3.438ms)
          Query Timeline: 7s745ms
             - Start execution: 66.767us (66.767us)
             - Planning finished: 6s108ms (6s108ms)
             - Ready to start 32 remote fragments: 6s111ms (2.565ms)
             - All 32 remote fragments started: 6s162ms (51.360ms)
             - Rows available: 7s694ms (1s531ms)
             - First row fetched: 7s728ms (34.364ms)
      

      During the plan generation this appears to be the hot path

      "Thread-8" prio=10 tid=0x00000000068d5800 nid=0xb1c5 runnable [0x00007fbc6b667000]
         java.lang.Thread.State: RUNNABLE
              at com.cloudera.impala.service.FeSupport.NativeEvalConstExprs(Native Method)
              at com.cloudera.impala.service.FeSupport.EvalConstExprs(FeSupport.java:172)
              at com.cloudera.impala.service.FeSupport.EvalPredicateBatch(FeSupport.java:211)
              at com.cloudera.impala.planner.HdfsPartitionFilter.getMatchingPartitionIds(HdfsPartitionFilter.java:94)
              at com.cloudera.impala.planner.HdfsPartitionPruner.evalPartitionFiltersInBe(HdfsPartitionPruner.java:452)
              at com.cloudera.impala.planner.HdfsPartitionPruner.prunePartitions(HdfsPartitionPruner.java:144)
              at com.cloudera.impala.planner.SingleNodePlanner.createHdfsScanPlan(SingleNodePlanner.java:1251)
              at com.cloudera.impala.planner.SingleNodePlanner.createScanNode(SingleNodePlanner.java:1311)
              at com.cloudera.impala.planner.SingleNodePlanner.createTableRefNode(SingleNodePlanner.java:1544)
              at com.cloudera.impala.planner.SingleNodePlanner.createTableRefsPlan(SingleNodePlanner.java:806)
              at com.cloudera.impala.planner.SingleNodePlanner.createSelectPlan(SingleNodePlanner.java:645)
              at com.cloudera.impala.planner.SingleNodePlanner.createQueryPlan(SingleNodePlanner.java:234)
              at com.cloudera.impala.planner.SingleNodePlanner.createSingleNodePlan(SingleNodePlanner.java:140)
              at com.cloudera.impala.planner.Planner.createPlan(Planner.java:60)
              at com.cloudera.impala.service.Frontend.createExecRequest(Frontend.java:945)
              at com.cloudera.impala.service.JniFrontend.createExecRequest(JniFrontend.java:141)
      

      Attachments

        1. ExplainPlanGoingThroughCodeGen.csv
          10 kB
          Mostafa Mokhtar

        Activity

          People

            kwho Michael Ho
            mmokhtar Mostafa Mokhtar
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: