Yes, ! means a negative match. See Netdata simple patterns.
@ilyam8 Still the container metrics are not shown. Here is our current search pattern:
search for cgroups in subpaths matching = !*/init.scope !*-qemu !*.libvirt-qemu !/init.scope !/system !/systemd !/user !/user.slice !/lxc/*/* /lxc.monitor.* !/lxc.payload/*/* /lxc.payload.* *
In /sys/fs/cgroup/devices I could see the lxc containers (those “payload” and “monitor” thing) but nothing shows up in Netdata.
Not sure how to proceed. If needed we can provide you shell access to our server so that you can go in and see what’s wrong.
Thank you.
Any update on this? It would be very helpful and appreciated if you could look into this again.
I’m not sure how different these both are. I thought maybe this is some configuration issue of lxc or maybe the version of LXD hypervisor we’re using. But even docker containers are not visible in the Netdata dashboard. Not sure why. I am running the container under the root user.
Hi, @philip. I don’t know what the problem is. There is no problem according to the logs. I will try to add more logging and create another custom image ~ on Friday.
Yes, it is ilyam8/netdata-test-for-github.
Ah, thanks for taking the time to help us. Very much appreciated.
So I followed the instructions which you stated previously.
The flood protection flag is also set to zero.
Here is the complete logs (the one I got from docker logs and after further piping):
The log is huge 
Thanks!
I see there are cgroups that discovered and cgroups.plugin created charts for them, e.g.
2023-02-02 14:15:49: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.hydrogen' (name 'hydrogen')
2023-02-02 14:15:49: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.planb' (name 'planb')
2023-02-02 14:15:49: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.microsrvc2' (name 'microsrvc2')
2023-02-02 14:15:49: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.mynode7' (name 'mynode7')
2023-02-02 14:15:49: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.mynode7/user.slice/user-0.slice/session-1328872.scope' (name 'mynode7_user.slice_user-0.slice_session-1328872.scope')
2023-02-02 14:15:49: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.sapphirecap4' (name 'sapphirecap4')
2023-02-02 14:15:49: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.sapphirecap4/docker' (name 'sapphirecap4_docker')
2023-02-02 14:15:49: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.emailserver' (name 'emailserver')
2023-02-02 14:15:49: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.reverseprox1' (name 'reverseprox1')
2023-02-02 14:15:49: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.butterflyeu' (name 'butterflyeu')
2023-02-02 14:15:49: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.peng' (name 'peng')
2023-02-02 14:15:49: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.bipradix' (name 'bipradix')
2023-02-02 14:15:49: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.ireallydo' (name 'ireallydo')
2023-02-02 14:15:49: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.ads' (name 'ads')
2023-02-02 14:15:49: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.kindao' (name 'kindao')
2023-02-02 14:15:49: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.viecondocker' (name 'viecondocker')
2023-02-02 14:15:49: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.viecondocker/docker' (name 'lxc.payload.viecondocker_docker')
2023-02-02 14:15:49: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.iran' (name 'iran')
2023-02-02 14:15:49: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.aclserver' (name 'aclserver')
2023-02-02 14:15:49: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.sapphirecap2' (name 'sapphirecap2')
2023-02-02 14:15:49: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.sapphirecap2/docker' (name 'sapphirecap2_docker')
2023-02-02 14:15:49: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.deloittefond' (name 'deloittefond')
2023-02-02 14:15:49: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.authservice' (name 'authservice')
2023-02-02 14:15:49: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.valtameri' (name 'valtameri')
2023-02-02 14:15:49: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.valtameri/docker' (name 'valtameri_docker')
2023-02-02 14:15:49: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.darpa' (name 'darpa')
2023-02-02 14:15:49: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.pbserver' (name 'pbserver')
2023-02-02 14:15:49: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.sapphirecapi' (name 'sapphirecapi')
2023-02-02 14:15:49: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.sapphirecapi/docker' (name 'sapphirecapi_docker')
2023-02-02 14:15:59: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.viecondocker/docker/0fc187f9b31b34df7cf0a030294e824edffbf4a2091c4512f873ce56951c0013' (name '0fc187f9b31b')
2023-02-02 14:15:59: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.viecondocker/docker/63c2a92d439b76a56af20589b81f3d3a9f5646b6bba983ca0c83cd60f4e1d533' (name '63c2a92d439b')
2023-02-02 14:15:59: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.viecondocker/docker/99548abb9bf65a1237b2345ee999f3df5f96d30998bab8a4322a12c78289bb1e' (name '99548abb9bf6')
2023-02-02 14:15:59: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.viecondocker/docker/0429770e6f4a7b4936b14164357c363863c884d393b5120adc839edf3fb2baea' (name '0429770e6f4a')
2023-02-02 14:15:59: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.viecondocker/docker/606fa4a789f2cc4e3107a5b20e6a8e51adcfd2a0d23ba405162da7b082dbf9af' (name '606fa4a789f2')
2023-02-02 14:15:59: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.viecondocker/docker/756ca5a2b04863517ae03d9cfa702b98b7efb6c8e4aa68afcfebe3a1efe1bc96' (name '756ca5a2b048')
2023-02-02 14:15:59: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.viecondocker/docker/7db90d71314997e225b74696a097aa6c2d9241c66fcd42f5ac44fdf14d73238b' (name '7db90d713149')
2023-02-02 14:15:59: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.viecondocker/docker/079f12833fd21b8649132ad80280d1c42658a6f6a4634e00d99678aa1254fcb8' (name '079f12833fd2')
2023-02-02 14:15:59: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.viecondocker/docker/05b83f4d72c83e8b1a63ff3fcf0a746e667070db1f7afa1d702bcc0d55b40593' (name '05b83f4d72c8')
2023-02-02 14:15:59: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.viecondocker/docker/28379d9c27c9b15f7f5250da720521b6184aba9d6c19870c0a338448facc8805' (name '28379d9c27c9')
2023-02-02 14:15:59: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.viecondocker/docker/72b72e639a12539a23e735d6ddfd0a6cfe9a40d6d2db92a94e74be672166675f' (name '72b72e639a12')
2023-02-02 14:15:59: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.viecondocker/docker/4b05d227e9ed211bb1d05f8ccbb66b3f497b2bae6244d89bfba64a1506d7d68e' (name '4b05d227e9ed')
2023-02-02 14:15:59: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.sapphirecapi/docker/568cb930570d5716425edf5d53e3f1a6f3cfe739c4eac0eb4539ea8fd88253e6' (name '568cb930570d')
2023-02-02 14:15:59: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.sapphirecapi/docker/bf562b2773b587898e2ba971bba5cdc0a2ceeed0e5a02dc993ff4a6b81070a33' (name 'bf562b2773b5')
2023-02-02 14:15:59: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.sapphirecapi/docker/8e31a1a164b389885964b857e39be8f8115c798ed1afcdcefd0a0b85a04f4fbc' (name '8e31a1a164b3')
2023-02-02 14:15:59: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.sapphirecapi/docker/ae1076e4026610576d562d7cc5a6fed02d4c92e87cc15118b88292e1d45382b4' (name 'ae1076e40266')
2023-02-02 14:15:59: netdata INFO : P[cgroups] : CGROUP READ: create CPU USAGE CHART for cgroup id '/lxc.payload.sapphirecapi/docker/1d5e15cb7ef5f810f3f47e7b8d74fee286e0f4432dbc236d6f2b7af51ee68daf' (name '1d5e15cb7ef5')
For those stats are shown according to the log.
A lot of filtered out because of max cgroups to allow limit (default is 1000). You can try to increase it.
No we do not see any charts, not even for those.
I’ve increased the cgroup limit to 5000 and restarted Netdata. Waited for a few minutes. Still container metrics are not shown.
An update: The docker image you shared started showing the LXC containers, before it did not (with the first image you built for us). But it didn’t show all of them. So I increased the cgroup limit inside the docker container and I think now all container metrics are shown.
However the netdata installed directly on the host doesn’t show container metrics at all, not even after increasing the cgroup limit to 5000.
Edit:
The version of Netdata.
Docker netdata: v1.37.0-248-gccc9b4bc1
host netdata: v1.37.1
Sorry for bothering you too much.
Any update on this? Quite not sure why same versions of Netdata are acting differently 
Hi, @philip. Unfortunately, I have nothing in mind to debug it further (i see no problems in logs).
- from the logs, I see that container charts were created. There is no problem.
- some cgroups get filtered because of
max cgroups to allow(expected, the option can be tunned).
@philip there’s definitely something strange going on, when the same Netdata version shows you two different results on the UI based on whether its running on the host, or containerized. Even more strange when the one showing more data is the containerized version. I’d be checking for permissions, but the log entries above clearly show that charts were created. So either the log is not from the correct netdata, or it’s something we’ve never seen before.
Let’s forget the containerized version for a bit and see if we can at least figure out if it’s an issue with the UI, or the charts really aren’t there.
- Go to [yournetdatahostui]/api/v1/charts and check that list for your lxc containers. If they are there but not on your UI, something very weird is happening. I expect they won’t be there either.
- Cleanly uninstall and remove any user configuration you may have applied. Ensure all netdata related directories are removed. Reinstall using kickstart. See what you get.
- Connect the node to netdata cloud and see if the charts appear there.
In the charts on the affected hosts I do not see any fields like “cgroup_{container name}”.cpu or something but you guys say that charts are created, not sure what to say.
Have done a reinstall. Will let you guys know how it goes.
The charts are not shown in the API endpoint data.
Reinstalled Netdata. Still the metrics are not shown.
Connected the host to the Netdata cloud. There too, no metrics of LXC containers.
How about if we suggest doing the verbose logging directly on our host? Could you give instructions on how to do that?
Thank you.
We can try that, but you will need to compile Netdata from source on that machine.
One thing missing from that doc I believe is that you need to compile it with debug flags on (correcting the document now).
Edit: changed the link, we have two documents for compile from source, I had linked the one for package maintainers.
Have installed Netdata from source with debugging enabled.
In the netdata.conf, have set
- debug flags = 0xffffffffffffffff to debug everything
- errors flood protection period = 0
- wiped out the existing error log using “cp /dev/null /var/log/netdata/error.log”
Here is the error and debug log.
debug log: debug.log - Google Drive
error log: error.log - Google Drive
Please let us know if you need further information.