We see a lot of emails with warnings for
cgroup_ram_in_use. We run Docker Swarm on 10+ bare metal servers with 100+ services and 400+ containers. Is this cgroup_ram_in_use meant for a whole server, for a single service or for a single container?
We are running prometheus/grafana but we can’t see RAM issues there. How can I identify the source for the warnings?
Hello @bluepuma77 and welcome into our community, This (cgroup_ram_in_use) topic may give you some insights about this alert.
TTBOMK you have two options:
- You either relax the threshold (by editing their configuration, conf-file: health.d/cgroups.conf) of the “warn” state for this alert. This will just hide the “issue”, so I don’t advise you to stick with this action.
- Set greater limits for the containers in question. Netdata informs you that these containers are running near the edge of their limits. Note here: If none limit is set; defaults apply by Docker but this shouldn’t be a case for all of your containers/workloads. Different containers, different demands.
I would suggest you to visit also the following topic:
Think about it and let us know about your situation
We are running prometheus/grafana but we can’t see RAM issues there.
Not so sure that they have predefined alerts for this resource.
The forum’s automated system blocked this post as spam. Sorry for the inconvenience. I can see some spikes that may have triggered this alert (I need to know which are the actual limits for this container), but it seems like an issue that this alert report is advertising that the triggered value is 57.52%. I will open ticket for this.
@Tasos_Katsoulas Looked at it again and identified the
b00... as the container ID, which we didn’t recognise before.
The container was cadvisor which had memory limits set to 128M/64M.
For us it would be very helpful to add the container name to the display of containers in the Netdata agent GUI and Netdata Cloud.
You should be able to see the container names, I believe that there are some extra steps to be done in your system to be able to see them. Best guess: Netdata dont have enough privileges to access the Docker sock. But let’s not guess.
What steps did you follow to install the Agent? Could you provide me with the output of the
netdata -W buildinfo and the platform specs (OS, kernel) for one of these nodes?
Extra step: is the netdata user in the docker’s user group? (
Running with Docker on Debian, no dedicated Netdata user on host. The local agent sees the containers with IDs, CPU, RAM, etc. but shows no names.
ID=$(hostname | sed 's/[^0-9]*//g')
docker run -d --name=netdata \
--env DISABLE_TELEMETRY=1 \
--hostname $(hostname) \
-p 10.0.0.$ID:19999:19999 \
-v netdataconfig:/etc/netdata \
-v netdatalib:/var/lib/netdata \
-v netdatacache:/var/cache/netdata \
-v /etc/passwd:/host/etc/passwd:ro \
-v /etc/group:/host/etc/group:ro \
-v /proc:/host/proc:ro \
-v /sys:/host/sys:ro \
-v /etc/os-release:/host/etc/os-release:ro \
--restart unless-stopped \
--cap-add SYS_PTRACE \
--security-opt apparmor=unconfined \
-e NETDATA_CLAIM_TOKEN=abc \
-e NETDATA_CLAIM_URL=https://app.netdata.cloud \
/ # netdata -W buildinfo
Version: netdata v1.35.0-209-nightly
Configure options: '--prefix=/usr' '--sysconfdir=/etc' '--localstatedir=/var' '--libexecdir=/usr/libexec' '--libdir=/usr/lib' '--with-zlib' '--with-math' '--with-user=netdata' '--without-bundled-protobuf' '--disable-dependency-tracking' '--disable-ebpf' 'CFLAGS=-O2 -pipe' 'LDFLAGS='
Install type: oci
Binary architecture: x86_64
Native HTTPS: YES
Netdata Cloud: YES
ACLK Next Generation: YES
ACLK-NG New Cloud Protocol: YES
ACLK Legacy: NO
TLS Host Verification: YES
Machine Learning: YES
Stream Compression: YES
protobuf: YES (system)
cgroup Network Tracking: YES
Xen VBD Error Tracking: NO
AWS Kinesis: NO
GCP PubSub: NO
Prometheus Remote Write: YES
Oh ok, you run Netdata as a Docker container. Netdata container needs a way to access the Docker socket of the host machine. You can follow this Install Netdata with Docker | Learn Netdata subsection in our docs to guide you through it.
@Tasos_Katsoulas You should probably add that piece of information to your Docker setup guide in Netdata Cloud: