[FLUME-3365] flume high rss memeory occupied - ASF JIRA

XML

Word

Printable

JSON

Type: Bug
Status: Open
Priority: Minor
Resolution: Unresolved
Affects Version/s: 1.6.0
Fix Version/s: None
Component/s: Sinks+Sources
Labels:
None
Environment:

Hide

UbuntuDescription: Ubuntu 16.04.6 LTSRelease: 16.04Codename: xenial

java_env:1.8.0_171

jvm options:

/usr/bin/java -Xms200m -Xmx200m -Dflume.root.logger=info,console -Dflume.monitoring.type=http -Dflume.monitoring.port=1235 -XX:NativeMemoryTracking=detail

Show
UbuntuDescription: Ubuntu 16.04.6 LTSRelease: 16.04Codename: xenial java_env:1.8.0_171 jvm options: /usr/bin/java -Xms200m -Xmx200m -Dflume.root.logger=info,console -Dflume.monitoring.type=http -Dflume.monitoring.port=1235 -XX:NativeMemoryTracking=detail

We use flume-client (taildir-source and avro-sink) to collect machine's application logs to another flume B (all collection),

Flume B send events to kafka.

With a long runtime . we found flume-client occupied rss memory large than memory by used -Xmx

we think its may cause System OOMKiller to kill another application (which had occupied the largest memory)

Why flume can occupied high rss memory with a long runtime?

We use pmap and NativeMemoryTracking. but the amount of them not equal. emmm