Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-13012

Completed queries write fails regularly under heavy load

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • Impala 4.4.0
    • Impala 4.4.0
    • Backend
    • None

    Description

      Under heavy test load (running EE tests), Impala regularly fails to write completed queries with errors like

      W0411 19:11:07.764967 32713 workload-management.cc:435] failed to write completed queries table="sys.impala_query_log" record_count="10001"
      W0411 19:11:07.764983 32713 workload-management.cc:437] AnalysisException: Exceeded the statement expression limit (250000)
      Statement has 370039 expressions.
      

      After a few attempts, it floods logs with an error for each query that could not be written

      E0411 19:11:24.646953 32713 workload-management.cc:376] could not write completed query table="sys.impala_query_log" query_id="3142ceb1380b58e6:715b83d900000000"
      

      This seems like poor default behavior. Options for addressing it:

      1. Decrease the default for query_log_max_queued. Inserts are pretty constant at 37 expressions per entry. I'm not sure why that isn't 49, since that's the number of columns we have; maybe some fields are frequently omitted. I would cap query_log_max_queued to statement_expression_limit / number_of_columns ~ 5100.
      2. Allow workload management to set statement_expression_limit higher using a similar formula. This may be relatively safe as the expressions are simple.
      3. Ideally we would skip expression parsing and construct TExecRequest directly, but that's a much larger effort.

      Attachments

        Issue Links

          Activity

            People

              MikaelSmith Michael Smith
              MikaelSmith Michael Smith
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: