Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-12984

Show inactivity of data exchanges in query profile

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • Distributed Exec
    • None

    Description

      Many-to-many data exchanges can be bottlenecked by hotspot receiver such scenario described in IMPALA-6692 or when data spilling happens in subset of backend. Ideally, this occurrences should be easily figured out in query profile. But triaging this kind of issue often requires correlation analysis of several counters in query profile. There are few ideas on how to improve this identification:

      1. Upon query completion, let coordinator do some profile analysis and print WARNING in query profile pointing at the skew. One group of EXCHANGE senders and receivers can only complete simultaneously since all receivers need to wait for EOS signal from all senders. Let say we take max of TotalNetworkSendTime from all senders and max of DataWaitTime from all receivers, a "mutual wait" time of min(TotalNetworkSendTime,DataWaitTime) can be used as indicator of how long the exchanges are waiting for query operators above them to progress.
      2. Add "Max Inactive" column in ExecSummary table. Existing "Avg Time" and "Max Time" are derived from RuntimeProfileBase::local_time_ns_. If ExecSummary also display maximum value of RuntimeProfileBase::inactive_timer_ of each query operator as "Max Inactive", we can then compare it against "Max Time" and figure out which exchange is mostly idle waiting. The calculation between local_time_ns, children_total_time, and inactive_timer can be seen here at https://github.com/apache/impala/blob/0721858/be/src/util/runtime-profile.cc#L935-L938 

      Attachments

        1. Profile with slow data exchanges.txt
          1.41 MB
          Manish Maheshwari

        Activity

          People

            Unassigned Unassigned
            rizaon Riza Suminto
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: