insane netdata memory usage

Hello @Pavel_Rekun ,

I installed a CentOS 9 VM now, and I could not recreate the crash issue.
Now about eBPF.plugin, it only uses huge sample of memory when integration with apps or cgroup is enabled, this is expected, because we need to store information for different PIDs running on system. Since last year we merged PRs to reduce the number of PIDs we are monitoring and right now all integrations are completely disabled by default.
Did you configure ebpf.plugin on your host? Or are you running with default options?

Best regards!

running with default settings except for the child config.
@ilyam8 i’ll try disabling the plugin and report back.

@Pavel_Rekun did it help?

OK - needed a few days to confirm - looks like it solved the issue. all 3 instances of netdata service are now using ~100MB.

Is this a known issue? should it be fixed on your end? or i just need to keep the plugin disabled?

Thanks for testing @Pavel_Rekun. Let’s do one more check and see if that is ebpf.plugin or something has been fixed in the latest master.

Can you please re-enable ebpf.plugin on one instance? And let’s double-check that you don’t have a non-default ebpf configuration ( does /opt/netdata/etc/netdata/ebpf.d.conf exists?).

Hello @Pavel_Rekun ,

Are you running the same netdata version on all hosts? Or are the children running stable version?

Best regards!

all running same os and same agent version and same child config.

Just tried it - went to 1gb 5 sec after startup… no custom config file

Hello @Pavel_Rekun ,

We started reworking eBPF and we made an issue describing all steps that we will do [Feat]/[BUG]: eBPF memory improvement · Issue #14477 · netdata/netdata · GitHub.

The first PR that was already made is bringing small improvements, but a bigger change will come with the second PR.

Best regards!

Not a specialist, but from what I remember - more CPU cores you have more memory ebpf.plugin will use (check the mem.kernel chart Percpu dimension). How many do you have?