Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
1.10.0
-
None
Description
Connections should have Status Analytics Model instances that are unique for each connection such that calculations for the model can be referred to or updated when new observations are available. Currently the model appears to be shared across connection objects which leads to erroneous results, especially when a connection does not have enough observations to update the model (since the older values of a previous connection may still be present).
This behavior can be seen when debugging is enabled for analytics and noting differences between a Timer Driven Thread execution of predictions vs the NiFi Web Server thread predictions for multiple connections (a flow with one connection would not exhibit this issue). The Time Driven thread may not retrieve enough connection status observations to refresh the model, and when that occurs, it may still contain calculations from a previous model and use that for predictions. Observers may see behavior where web server and timer driven predictions only have similar or matching predictions for one connection but others may be mismatched, such as in the below (where connection BBBB has similar prediction between threads but AAAA, and CCCC do not):
2019-10-22 08:01:34,539 INFO [NiFi Web Server-23] o.a.nifi.reporting.StandardEventAccess >>>> AAAA: predicted time to COUNT backpressure = 00:44:13 / 2653416 2019-10-22 08:01:34,539 INFO [NiFi Web Server-23] o.a.nifi.reporting.StandardEventAccess >>>> AAAA: predicted time to BYTES backpressure = 01:09:13 / 4153511 2019-10-22 08:01:34,540 INFO [NiFi Web Server-23] o.a.nifi.reporting.StandardEventAccess >>>> CCCC: predicted time to COUNT backpressure = 00:32:31 / 1951609 2019-10-22 08:01:34,540 INFO [NiFi Web Server-23] o.a.nifi.reporting.StandardEventAccess >>>> CCCC: predicted time to BYTES backpressure = 00:32:28 / 1948504 2019-10-22 08:01:34,541 INFO [NiFi Web Server-23] o.a.nifi.reporting.StandardEventAccess >>>> BBBB: predicted time to COUNT backpressure = 07:29:14 / 26954837 2019-10-22 08:01:34,541 INFO [NiFi Web Server-23] o.a.nifi.reporting.StandardEventAccess >>>> BBBB: predicted time to BYTES backpressure = 05:24:56 / 19496954 2019-10-22 08:02:02,767 INFO [Timer-Driven Process Thread-2] o.a.nifi.reporting.StandardEventAccess >>>> AAAA: predicted time to COUNT backpressure = 04:08:36 / 14916150 2019-10-22 08:02:02,767 INFO [Timer-Driven Process Thread-2] o.a.nifi.reporting.StandardEventAccess >>>> AAAA: predicted time to BYTES backpressure = 04:17:49 / 15469243 2019-10-22 08:02:02,769 INFO [Timer-Driven Process Thread-2] o.a.nifi.reporting.StandardEventAccess >>>> CCCC: predicted time to COUNT backpressure = 01:13:42 / 4422829 2019-10-22 08:02:02,769 INFO [Timer-Driven Process Thread-2] o.a.nifi.reporting.StandardEventAccess >>>> CCCC: predicted time to BYTES backpressure = 00:57:15 / 3435109 2019-10-22 08:02:02,770 INFO [Timer-Driven Process Thread-2] o.a.nifi.reporting.StandardEventAccess >>>> BBBB: predicted time to COUNT backpressure = 07:28:46 / 26926608 2019-10-22 08:02:02,770 INFO [Timer-Driven Process Thread-2] o.a.nifi.reporting.StandardEventAccess >>>> BBBB: predicted time to BYTES backpressure = 05:24:28 / 19468725
A workaround for this is to increase the nifi.analytics.query.interval property or decrease the nifi.components.status.snapshot.frequency property to ensure obtaining enough observations for a refresh. However an appropriate fix is to ensure model instances are unique for each connection and to ensure that, by default, all threads will obtain enough observations for predictions.
Attachments
Issue Links
- links to
