LXC Containers Stats Are Not Shown

Yes, ! means a negative match. See Netdata simple patterns.

@ilyam8 Still the container metrics are not shown. Here is our current search pattern:

search for cgroups in subpaths matching = !*/init.scope  !*-qemu  !*.libvirt-qemu  !/init.scope  !/system  !/systemd  !/user  !/user.slice  !/lxc/*/*  /lxc.monitor.*  !/lxc.payload/*/*  /lxc.payload.*  *

@ilyam8

In /sys/fs/cgroup/devices I could see the lxc containers (those “payload” and “monitor” thing) but nothing shows up in Netdata.

Not sure how to proceed. If needed we can provide you shell access to our server so that you can go in and see what’s wrong.

Thank you.

@ilyam8

Any update on this? It would be very helpful and appreciated if you could look into this again.

@ilyam8

I’m not sure how different these both are. I thought maybe this is some configuration issue of lxc or maybe the version of LXD hypervisor we’re using. But even docker containers are not visible in the Netdata dashboard. Not sure why. I am running the container under the root user.

Hi, @philip. I don’t know what the problem is. There is no problem according to the logs. I will try to add more logging and create another custom image ~ on Friday.

@ilyam8

Just wanted to ask: Did you build that image?

Thanks.

Yes, it is ilyam8/netdata-test-for-github.

Ah, thanks for taking the time to help us. Very much appreciated.

So I followed the instructions which you stated previously.

The flood protection flag is also set to zero.

Here is the complete logs (the one I got from docker logs and after further piping):

The log is huge :slight_smile:

Thanks!

I see there are cgroups that discovered and cgroups.plugin created charts for them, e.g.

2023-02-02 14:15:49: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.hydrogen' (name 'hydrogen')
2023-02-02 14:15:49: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.planb' (name 'planb')
2023-02-02 14:15:49: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.microsrvc2' (name 'microsrvc2')
2023-02-02 14:15:49: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.mynode7' (name 'mynode7')
2023-02-02 14:15:49: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.mynode7/user.slice/user-0.slice/session-1328872.scope' (name 'mynode7_user.slice_user-0.slice_session-1328872.scope')
2023-02-02 14:15:49: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.sapphirecap4' (name 'sapphirecap4')
2023-02-02 14:15:49: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.sapphirecap4/docker' (name 'sapphirecap4_docker')
2023-02-02 14:15:49: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.emailserver' (name 'emailserver')
2023-02-02 14:15:49: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.reverseprox1' (name 'reverseprox1')
2023-02-02 14:15:49: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.butterflyeu' (name 'butterflyeu')
2023-02-02 14:15:49: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.peng' (name 'peng')
2023-02-02 14:15:49: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.bipradix' (name 'bipradix')
2023-02-02 14:15:49: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.ireallydo' (name 'ireallydo')
2023-02-02 14:15:49: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.ads' (name 'ads')
2023-02-02 14:15:49: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.kindao' (name 'kindao')
2023-02-02 14:15:49: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.viecondocker' (name 'viecondocker')
2023-02-02 14:15:49: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.viecondocker/docker' (name 'lxc.payload.viecondocker_docker')
2023-02-02 14:15:49: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.iran' (name 'iran')
2023-02-02 14:15:49: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.aclserver' (name 'aclserver')
2023-02-02 14:15:49: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.sapphirecap2' (name 'sapphirecap2')
2023-02-02 14:15:49: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.sapphirecap2/docker' (name 'sapphirecap2_docker')
2023-02-02 14:15:49: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.deloittefond' (name 'deloittefond')
2023-02-02 14:15:49: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.authservice' (name 'authservice')
2023-02-02 14:15:49: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.valtameri' (name 'valtameri')
2023-02-02 14:15:49: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.valtameri/docker' (name 'valtameri_docker')
2023-02-02 14:15:49: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.darpa' (name 'darpa')
2023-02-02 14:15:49: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.pbserver' (name 'pbserver')
2023-02-02 14:15:49: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.sapphirecapi' (name 'sapphirecapi')
2023-02-02 14:15:49: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.sapphirecapi/docker' (name 'sapphirecapi_docker')
2023-02-02 14:15:59: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.viecondocker/docker/0fc187f9b31b34df7cf0a030294e824edffbf4a2091c4512f873ce56951c0013' (name '0fc187f9b31b')
2023-02-02 14:15:59: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.viecondocker/docker/63c2a92d439b76a56af20589b81f3d3a9f5646b6bba983ca0c83cd60f4e1d533' (name '63c2a92d439b')
2023-02-02 14:15:59: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.viecondocker/docker/99548abb9bf65a1237b2345ee999f3df5f96d30998bab8a4322a12c78289bb1e' (name '99548abb9bf6')
2023-02-02 14:15:59: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.viecondocker/docker/0429770e6f4a7b4936b14164357c363863c884d393b5120adc839edf3fb2baea' (name '0429770e6f4a')
2023-02-02 14:15:59: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.viecondocker/docker/606fa4a789f2cc4e3107a5b20e6a8e51adcfd2a0d23ba405162da7b082dbf9af' (name '606fa4a789f2')
2023-02-02 14:15:59: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.viecondocker/docker/756ca5a2b04863517ae03d9cfa702b98b7efb6c8e4aa68afcfebe3a1efe1bc96' (name '756ca5a2b048')
2023-02-02 14:15:59: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.viecondocker/docker/7db90d71314997e225b74696a097aa6c2d9241c66fcd42f5ac44fdf14d73238b' (name '7db90d713149')
2023-02-02 14:15:59: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.viecondocker/docker/079f12833fd21b8649132ad80280d1c42658a6f6a4634e00d99678aa1254fcb8' (name '079f12833fd2')
2023-02-02 14:15:59: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.viecondocker/docker/05b83f4d72c83e8b1a63ff3fcf0a746e667070db1f7afa1d702bcc0d55b40593' (name '05b83f4d72c8')
2023-02-02 14:15:59: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.viecondocker/docker/28379d9c27c9b15f7f5250da720521b6184aba9d6c19870c0a338448facc8805' (name '28379d9c27c9')
2023-02-02 14:15:59: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.viecondocker/docker/72b72e639a12539a23e735d6ddfd0a6cfe9a40d6d2db92a94e74be672166675f' (name '72b72e639a12')
2023-02-02 14:15:59: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.viecondocker/docker/4b05d227e9ed211bb1d05f8ccbb66b3f497b2bae6244d89bfba64a1506d7d68e' (name '4b05d227e9ed')
2023-02-02 14:15:59: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.sapphirecapi/docker/568cb930570d5716425edf5d53e3f1a6f3cfe739c4eac0eb4539ea8fd88253e6' (name '568cb930570d')
2023-02-02 14:15:59: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.sapphirecapi/docker/bf562b2773b587898e2ba971bba5cdc0a2ceeed0e5a02dc993ff4a6b81070a33' (name 'bf562b2773b5')
2023-02-02 14:15:59: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.sapphirecapi/docker/8e31a1a164b389885964b857e39be8f8115c798ed1afcdcefd0a0b85a04f4fbc' (name '8e31a1a164b3')
2023-02-02 14:15:59: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.sapphirecapi/docker/ae1076e4026610576d562d7cc5a6fed02d4c92e87cc15118b88292e1d45382b4' (name 'ae1076e40266')
2023-02-02 14:15:59: netdata INFO  : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.sapphirecapi/docker/1d5e15cb7ef5f810f3f47e7b8d74fee286e0f4432dbc236d6f2b7af51ee68daf' (name '1d5e15cb7ef5')

For those stats are shown according to the log.


A lot of filtered out because of max cgroups to allow limit (default is 1000). You can try to increase it.

@ilyam8

No we do not see any charts, not even for those.

I’ve increased the cgroup limit to 5000 and restarted Netdata. Waited for a few minutes. Still container metrics are not shown.

@ilyam8

An update: The docker image you shared started showing the LXC containers, before it did not (with the first image you built for us). But it didn’t show all of them. So I increased the cgroup limit inside the docker container and I think now all container metrics are shown.

However the netdata installed directly on the host doesn’t show container metrics at all, not even after increasing the cgroup limit to 5000.

Edit:

The version of Netdata.

Docker netdata: v1.37.0-248-gccc9b4bc1
host netdata: v1.37.1