I have several virtual servers with only 512 MB RAM. Netdata is taking up a pretty big chunk of the RAM (~20%, around 104 MB RSS). What’s the best way to reduce the memory while still collecting metrics frequently? The disk is relatively fast (NVMe drive) and I have several GB of free space so it’s fine to trade off memory for disk, however I saw that the map storage engine is not recommended on nodes that run other applications.
I’ve been using Netdata for a long time, from before it had the dbengine, which I thought would help with this. I’m using the default Netdata config, which I assume would be using the dbengine now. I thought that reducing the page cache size might help, but I saw on How to optimize the Netdata Agent's performance | Learn Netdata that it has a minimum of 32 MB, which is what mine is currently set to.
Any recommendations for this environment? Should I just use streaming rather than attempting to store the data on the nodes themselves?
Welcome to our forum and thank you for using Netdata !!
Netdata’s dbengine keeps metrics on disk but requires memory to keep track of the stored metrics (index).
This index grows as data retention is increased and needs to be in RAM. This is different from the page cache and cannot be avoided at this point.
The index is built in memory during agent restart and during runtime as metrics are collected. A large retention period will have netdata’s RSS increase up to the point where metric retention occurs. Having plenty of storage in your case will not help to offset the RAM usage.
I would suggest to use streaming (to a more powerful machine) or reduce the metric retention.
Streaming seems to use a non-trivial amount of bandwidth (seems like around 0.4 Mb/s with default settings) with makes it less than ideal. I guess I can reduce that by decreasing the granularity to reduce the amount of traffic used by streaming. Alternatively I’ll try dropping the retention on the low RAM machines.