Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-8656 Support for eagerly fetching and spooling all query result rows
  3. IMPALA-8803

Coordinator should release admitted memory per-backend rather than per-query

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • Impala 3.4.0
    • None
    • None
    • ghx-label-2

    Description

      When SPOOL_QUERY_RESULTS is true, the coordinator backend may be long lived, even though all other backends for the query have completed. Currently, the Coordinator only releases admitted memory when the entire query has completed (include the coordinator fragment) - https://github.com/apache/impala/blob/72c9370856d7436885adbee3e8da7e7d9336df15/be/src/runtime/coordinator.cc#L562

      In order to more aggressively return admitted memory, the coordinator should release memory when each backend for a query completes, rather than waiting for the entire query to complete.

      Releasing memory per backend should be batched because releasing admitted memory in the admission controller requires obtaining a global lock and refreshing the internal stats of the admission controller. Batching will help mitigate any additional overhead from releasing admitted memory per backend.

      Attachments

        Issue Links

          Activity

            People

              stakiar Sahil Takiar
              stakiar Sahil Takiar
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: