Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-17063

Do not buffer raw log params in LogWatcher

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • main (10.0), 9.4
    • 9.5
    • None
    • None

    Description

      As mkhl suggested on SOLR-13315, it would be possible for LogWatcher, rather than buffering all the raw parameters of a log message, to pre-compute the derivative {{SolrDocument}}s and buffer those instead.

      SOLR-13315 ended up being addressed in a different way, but "memory-leak-like" behavior is still possible and I think we should follow mkhl's suggestion. The problem we ran into recently was a huge distributed query, dispersed to lots of shards. One of the ShardRequests failed, causing the failed ShardResponse to be logged at warn level in SearchHandler:

      log.warn("Shard request failed : {}", srsp);
      

      The logged ShardResponse indirectly holds a reference (via ShardRequest ref) to all the other ShardResponses, some of which succeeded and were quite large. These references then just sit in LogWatcher's history CircularBuffer until they are overwritten by subsequent log messages (default WARN/ERROR level). So as long as you have fewer than 50 (default config) subsequent WARN/ERROR level messages, these references will be held (in principle forever). In our case it consumed a substantial portion of the heap.

      This issue proposes that LogWatcher should process LogEvents into SolrDocuments (the form they take upon retrieval) when they are initially added to the history buffer. The downside is some minimal amount of extra processing in the case where the events are never retrieved from the buffer; the upside is that the params don't needlessly retain heap.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              magibney Michael Gibney
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m