Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-8656 Support for eagerly fetching and spooling all query result rows
  3. IMPALA-8780

Implementation of BufferedPlanRootSink where FlushFinal blocks until all rows are fetched

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • Impala 3.3.0
    • Backend
    • None
    • ghx-label-5

    Description

      Implement BufferedPlanRootSink so that FlushFinal blocks until all rows are fetched. The implementation should use the RowBatchQueue introduced by IMPALA-8779. By blocking in FlushFinal all non-coordinator fragments will be closed if all results fit in the RowBatchQueue. BufferedPlanRootSink::Send should enqueue each given RowBatch onto the queue and then return. If the queue is full, it should block until there is more space left in the queue. BufferedPlanRootSink::GetNext reads from the queue and then fills in the given QueryResultSet by using the DataSink ScalarExprEvaluator-s. Since the producer thread can call BufferedPlanRootSink::Close while the consumer is calling BufferedPlanRootSink::GetNext the two methods need to be synchronized so that the DataSink MemTracker-s are not closed while GetNext is running.

      The implementation of BufferedPlanRootSink should remain the same regardless of whether a std::queue backed RowBatchQueue or a BufferedTupleStream backed RowBatchQueue is used.

      BufferedPlanRootSink and BlockingPlanRootSink are similar in the sense that BlockingPlanRootSink buffers one RowBatch, so for queries that return under 1024 rows, all non-coordinator fragments are closed immediately as well. The advantage of BufferedPlanRootSink is that allows buffering for 1+ RowBatch-es.

      Attachments

        Activity

          People

            stakiar Sahil Takiar
            stakiar Sahil Takiar
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: