Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-10883

Dense runtime profile missing some fragment instance profile

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • Impala 4.0.0
    • Impala 4.0.1
    • Distributed Exec
    • None
    • ghx-label-1

    Description

      I ran TPC-DS Q78 in the following setup:

      • Cluster of 20 nodes
      • MT_DOP=12
      • --gen_experimental_profile=true

      When I check the query profile of the completed query, I notice that couple fragment are missing profile from some instances. This missing profile is indicated in the missing instances id like this:

      Fragment F00 [228 instances]:
 Instances: Instance 204c719f48777536:b83eb0ed00000001 (host=ia0306.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed00000002 (host=ia0306.halxg.cloudera.com:27000), , Instance 204c719f48777536:b83eb0ed00000004 (host=ia0306.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed00000005 (host=ia0306.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed00000006 (host=ia0306.halxg.cloudera.com:27000), , , Instance 204c719f48777536:b83eb0ed00000009 (host=ia0306.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed0000000a (host=ia0306.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed0000000b (host=ia0306.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed0000000c (host=ia0306.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed0000000d (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed0000000e (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed0000000f (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed00000010 (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed00000011 (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed00000012 (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed00000013 (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed00000014 (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed00000015 (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed00000016 (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed00000017 (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed00000018 (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed00000019 (host=ia0322.halxg.cloudera.com:27000), , , , Instance 204c719f48777536:b83eb0ed0000001d (host=ia0322.halxg.cloudera.com:27000), , , , Instance 204c719f48777536:b83eb0ed00000021 (host=ia0322.halxg.cloudera.com:27000), , , , Instance 204c719f48777536:b83eb0ed00000025 (host=ia0317.halxg.cloudera.com:27000), , , , ...

      Note several missing instances from host ia0306, ia0322, ia0317, and so on.

      I tried to tune FLAGS_status_report_interval_ms faster to every 500ms and there are even more instance profile missing. On the other hand, setting FLAGS_status_report_interval_ms=0 (only send final report) will result in complete runtime profile.

      Attachments

        Issue Links

          relates to

          Sub-task - The sub-task of the issue IMPALA-9846 Switch to aggregated runtime profile representation

          • Major - Major loss of function.
          • Open

          Activity

            People

              rizaon Riza Suminto
              rizaon Riza Suminto
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: