What are the reasonable limits for Netdata? You have one parent handling streaming data from multiple nodes. There has to be some limit. I have a cluster with 130+ nodes with 7K+ pods and it seems Netdata parent is really struggling. Is there any documentation on how much data a Netdata parent can reasonably handle?
I don’t believe versions of Netdata after v1.31 or v1.33 can handle 100+ clients with cgroups on and each server hosting 50+ pods. At least that’s what I see. I tried adding the config:
accept a streaming request every seconds = 1
That just makes it completely unusable. The clients never get a chance to send the data.
If I undo that setting then the panel that shows workers for streamrcv shows that all those workers get busy and then I have huge gaps in the panels, if there’s any data at all.
You should update the docs and mention that the Netdata Parent can handle only N amount of data.
I needed to set this in the config. Things seem to be working ok now.
web server threads = 12