Uploaded image for project: 'Sling'
  1. Sling
  2. SLING-9078

Keep skipped offsets in memory

    XMLWordPrintableJSON

Details

    Description

      In SLING-8908 we introduced a mechanism to keep track of skipped packages such that in case of journal availability flip flop, the agents would still progress. The implementation was done by persisting the offset in the repository. In environments where many skipped packages will occur (e.g. large customer with multiple agents) then those commits will start to generate a non negligible load on the repository, consuming resources and creating bottlenecks.

       

      A first mitigation to this issue has been added in SLING-8965, however for very large customers the mitigation will not be effective. Increasing the grouping of skipped packages (e.g. 1/100 instead of 1/10) reduce the load on the repository but may fail to ensure progress in case of flip/flop.

       

      Instead of storing the skipped packages in the repository, we should keep them in memory. A new service that does not reference the JournalAvailable service could be added. The offsets would be consolidated on demand from the stored offsets and those stored in memory. 

      Attachments

        Activity

          People

            Unassigned Unassigned
            marett Timothee Maret
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: