insane netdata memory usage

Thiago_Marques_0 · January 18, 2023, 6:34pm

I installed a CentOS 9 VM now, and I could not recreate the crash issue.
Now about eBPF.plugin, it only uses huge sample of memory when integration with apps or cgroup is enabled, this is expected, because we need to store information for different PIDs running on system. Since last year we merged PRs to reduce the number of PIDs we are monitoring and right now all integrations are completely disabled by default.
Did you configure ebpf.plugin on your host? Or are you running with default options?

Best regards!

Pavel_Rekun · January 19, 2023, 8:34am

running with default settings except for the child config.
@ilyam8 i’ll try disabling the plugin and report back.

ilyam8 · January 22, 2023, 12:32pm

@Pavel_Rekun did it help?

Pavel_Rekun · January 23, 2023, 9:17am

OK - needed a few days to confirm - looks like it solved the issue. all 3 instances of netdata service are now using ~100MB.

Is this a known issue? should it be fixed on your end? or i just need to keep the plugin disabled?

ilyam8 · January 23, 2023, 12:02pm

Thanks for testing @Pavel_Rekun. Let’s do one more check and see if that is ebpf.plugin or something has been fixed in the latest master.

Can you please re-enable ebpf.plugin on one instance? And let’s double-check that you don’t have a non-default ebpf configuration ( does /opt/netdata/etc/netdata/ebpf.d.conf exists?).

Thiago_Marques_0 · January 23, 2023, 12:02pm

Hello @Pavel_Rekun ,

Are you running the same netdata version on all hosts? Or are the children running stable version?

Best regards!

Pavel_Rekun · January 23, 2023, 1:42pm

all running same os and same agent version and same child config.

Pavel_Rekun · January 23, 2023, 1:45pm

Just tried it - went to 1gb 5 sec after startup… no custom config file

Thiago_Marques_0 · February 9, 2023, 2:39pm

Hello @Pavel_Rekun ,

We started reworking eBPF and we made an issue describing all steps that we will do [Feat]/[BUG]: eBPF memory improvement · Issue #14477 · netdata/netdata · GitHub.

The first PR that was already made is bringing small improvements, but a bigger change will come with the second PR.

Best regards!

ilyam8 · February 9, 2023, 7:05pm

Not a specialist, but from what I remember - more CPU cores you have more memory ebpf.plugin will use (check the mem.kernel chart Percpu dimension). How many do you have?