cgroup_ram_in_use warnings with Docker Swarm / services / containers

Help
1

We see a lot of emails with warnings for cgroup_ram_in_use. We run Docker Swarm on 10+ bare metal servers with 100+ services and 400+ containers. Is this cgroup_ram_in_use meant for a whole server, for a single service or for a single container?

We are running prometheus/grafana but we can’t see RAM issues there. How can I identify the source for the warnings?

2

Netdata-Cloud-Cgroup-RAM
Netdata-Cloud-Cgroup-RAM2180×1120 258 KB

3

Hello @bluepuma77 and welcome into our community, This (cgroup_ram_in_use) topic may give you some insights about this alert.

TTBOMK you have two options:

  1. You either relax the threshold (by editing their configuration, conf-file: health.d/cgroups.conf) of the “warn” state for this alert. This will just hide the “issue”, so I don’t advise you to stick with this action.
  2. Set greater limits for the containers in question. Netdata informs you that these containers are running near the edge of their limits. Note here: If none limit is set; defaults apply by Docker but this shouldn’t be a case for all of your containers/workloads. Different containers, different demands.

I would suggest you to visit also the following topic:

Think about it and let us know about your situation

We are running prometheus/grafana but we can’t see RAM issues there.

Not so sure that they have predefined alerts for this resource.

4

The forum’s automated system blocked this post as spam. Sorry for the inconvenience. I can see some spikes that may have triggered this alert (I need to know which are the actual limits for this container), but it seems like an issue that this alert report is advertising that the triggered value is 57.52%. I will open ticket for this.

5

@Tasos_Katsoulas Looked at it again and identified the b00... as the container ID, which we didn’t recognise before.

The container was cadvisor which had memory limits set to 128M/64M.

For us it would be very helpful to add the container name to the display of containers in the Netdata agent GUI and Netdata Cloud.

6

You should be able to see the container names, I believe that there are some extra steps to be done in your system to be able to see them. Best guess: Netdata dont have enough privileges to access the Docker sock. But let’s not guess.

What steps did you follow to install the Agent? Could you provide me with the output of the netdata -W buildinfo and the platform specs (OS, kernel) for one of these nodes?

Extra step: is the netdata user in the docker’s user group? (id netdata)

Tasos.

7

Running with Docker on Debian, no dedicated Netdata user on host. The local agent sees the containers with IDs, CPU, RAM, etc. but shows no names.

ID=$(hostname | sed 's/[^0-9]*//g')

docker run -d --name=netdata \
  --env DISABLE_TELEMETRY=1 \
  --hostname $(hostname) \
  -p 10.0.0.$ID:19999:19999 \
  -v netdataconfig:/etc/netdata \
  -v netdatalib:/var/lib/netdata \
  -v netdatacache:/var/cache/netdata \
  -v /etc/passwd:/host/etc/passwd:ro \
  -v /etc/group:/host/etc/group:ro \
  -v /proc:/host/proc:ro \
  -v /sys:/host/sys:ro \
  -v /etc/os-release:/host/etc/os-release:ro \
  --restart unless-stopped \
  --cap-add SYS_PTRACE \
  --security-opt apparmor=unconfined \
  -e NETDATA_CLAIM_TOKEN=abc \
  -e NETDATA_CLAIM_URL=https://app.netdata.cloud \
  netdata/netdata

/ # netdata -W buildinfo
Version: netdata v1.35.0-209-nightly
Configure options:  '--prefix=/usr' '--sysconfdir=/etc' '--localstatedir=/var' '--libexecdir=/usr/libexec' '--libdir=/usr/lib' '--with-zlib' '--with-math' '--with-user=netdata' '--without-bundled-protobuf' '--disable-dependency-tracking' '--disable-ebpf' 'CFLAGS=-O2 -pipe' 'LDFLAGS='
Install type: oci
    Binary architecture: x86_64
Features:
    dbengine:                   YES
    Native HTTPS:               YES
    Netdata Cloud:              YES
    ACLK Next Generation:       YES
    ACLK-NG New Cloud Protocol: YES
    ACLK Legacy:                NO
    TLS Host Verification:      YES
    Machine Learning:           YES
    Stream Compression:         YES
Libraries:
    protobuf:                YES (system)
    jemalloc:                NO
    JSON-C:                  YES
    libcap:                  NO
    libcrypto:               YES
    libm:                    YES
    tcalloc:                 NO
    zlib:                    YES
Plugins:
    apps:                    YES
    cgroup Network Tracking: YES
    CUPS:                    NO
    EBPF:                    NO
    IPMI:                    YES
    NFACCT:                  NO
    perf:                    YES
    slabinfo:                YES
    Xen:                     NO
    Xen VBD Error Tracking:  NO
Exporters:
    AWS Kinesis:             NO
    GCP PubSub:              NO
    MongoDB:                 YES
    Prometheus Remote Write: YES

Netdata-agent
Netdata-agent1920×1048 95.2 KB

8

Oh ok, you run Netdata as a Docker container. Netdata container needs a way to access the Docker socket of the host machine. You can follow this Install Netdata with Docker | Learn Netdata subsection in our docs to guide you through it.

9

@Tasos_Katsoulas You should probably add that piece of information to your Docker setup guide in Netdata Cloud:

Netdata-Cloud-Docker-Setup
Netdata-Cloud-Docker-Setup1418×1628 269 KB