Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-6856

High memory estimates in Impala 2.11

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • Impala 2.11.0
    • None
    • Frontend
    • ghx-label-2

    Description

      Upgraded Impala cluster recently from 2.5 to 2.11 through CDH and observed that for some queries, memory estimates are very high now compared to pre upgrade days. Attached 1 such query whose memory estimates used to be 13-15 GB before started showing as 256 GB after the upgrade and not even entering the pool as it is directly getting rejected because of max pool size. As a temp fix, We have set mem limit for those queries and managing it for now.

      In addition, also seeing different memory estimates for different users belonging to different pools for the same query. Does pool max size plays an any role in memory estimates? Surprisingly, observed this behaviour on 1st day of upgrade. Tried to reproduce now, but it is showing same memory estimates for different users.

      Attachments

        1. query_high_mem_post_upgrade.sql
          24 kB
          Manikandan R
        2. query_high_mem_post_upgrade_profile.txt
          553 kB
          Manikandan R

        Issue Links

          Activity

            People

              Unassigned Unassigned
              manirajv06@gmail.com Manikandan R
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated: