Hello @ilyam8
I’ve disabled it, as you said, set it to zero (before it was 1200).
With cgroup, there was no other logs. The ones I posted were the only cgroup logs, no more after 11:14:23
As to Docker think you said: Do you mean you’ll generate a custom docker image for us so that we can deploy Netdata on our systems using Docker?
As you asked, I emailed the snapshot to you.
Thank you very much for your assistance!
Yes, just to test. I built a Netdata image with debugging.
netdata.conf with the following content:[logs]
debug flags = 0x0000000000100000
docker run -d --name=netdata_test_cgroups \
-p 20000:19999 \
-v netdatacache_test_cgroups:/var/cache/netdata \
-v /etc/passwd:/host/etc/passwd:ro \
-v /etc/group:/host/etc/group:ro \
-v /proc:/host/proc:ro \
-v /sys:/host/sys:ro \
-v $(pwd)/netdata.conf:/etc/netdata/netdata.conf \
-v /etc/os-release:/host/etc/os-release:ro \
--restart unless-stopped \
--cap-add SYS_PTRACE \
--security-opt apparmor=unconfined \
ilyam8/netdata-test-for-github
docker logs netdata_test_cgroups 2>&1 | grep -i cgroup | grep -v "UUID"
When done, to remove
docker stop netdata_test_cgroups
docker rm netdata_test_cgroups
docker volume rm netdatacache_test_cgroups
docker rmi ilyam8/netdata-test-for-github:latest
Here is the logs you requested. I’ve uploaded it to Google Drive as pastebin too limit the content size to 512KB.
Can you also share the tree -d /sys/fs/cgroup/ output?
Interesting, and you don’t see the container metrics?
It looks like all lxc.payload.X groups have been found and are being collected.
Downloads $ grep "lxc\.payload\..*" netdata_docker_logs.txt | grep "is called"
2022-12-15 10:39:27: cgroup-name.sh: INFO: cgroup 'lxc.payload.hydrogen' is called 'hydrogen'
2022-12-15 10:39:28: cgroup-name.sh: INFO: cgroup 'lxc.payload.planb' is called 'planb'
2022-12-15 10:39:30: cgroup-name.sh: INFO: cgroup 'lxc.payload.microsrvc2' is called 'microsrvc2'
2022-12-15 10:39:32: cgroup-name.sh: INFO: cgroup 'lxc.payload.mynode7' is called 'mynode7'
2022-12-15 10:39:37: cgroup-name.sh: INFO: cgroup 'lxc.payload.sapphirecap4' is called 'sapphirecap4'
2022-12-15 10:39:39: cgroup-name.sh: INFO: cgroup 'lxc.payload.emailserver' is called 'emailserver'
2022-12-15 10:39:41: cgroup-name.sh: INFO: cgroup 'lxc.payload.reverseprox1' is called 'reverseprox1'
2022-12-15 10:39:43: cgroup-name.sh: INFO: cgroup 'lxc.payload.butterflyeu' is called 'butterflyeu'
2022-12-15 10:39:49: cgroup-name.sh: INFO: cgroup 'lxc.payload.peng' is called 'peng'
2022-12-15 10:39:51: cgroup-name.sh: INFO: cgroup 'lxc.payload.bipradix' is called 'bipradix'
2022-12-15 10:39:52: cgroup-name.sh: INFO: cgroup 'lxc.payload.ireallydo' is called 'ireallydo'
2022-12-15 10:39:54: cgroup-name.sh: INFO: cgroup 'lxc.payload.ads' is called 'ads'
...
Downloads $ grep "lxc\.payload\..*" netdata_docker_logs.txt | grep "reading metrics for cgroups"
2022-12-15 10:43:31: netdata DEBUG : PLUGIN[cgroups] : (1613@collectors/cgroups.p:read_cgroup ): reading metrics for cgroups '/lxc.payload.hydrogen'
2022-12-15 10:43:31: netdata DEBUG : PLUGIN[cgroups] : (1613@collectors/cgroups.p:read_cgroup ): reading metrics for cgroups '/lxc.payload.planb'
2022-12-15 10:43:31: netdata DEBUG : PLUGIN[cgroups] : (1613@collectors/cgroups.p:read_cgroup ): reading metrics for cgroups '/lxc.payload.microsrvc2'
2022-12-15 10:43:31: netdata DEBUG : PLUGIN[cgroups] : (1613@collectors/cgroups.p:read_cgroup ): reading metrics for cgroups '/lxc.payload.mynode7'
2022-12-15 10:43:31: netdata DEBUG : PLUGIN[cgroups] : (1613@collectors/cgroups.p:read_cgroup ): reading metrics for cgroups '/lxc.payload.sapphirecap4'
2022-12-15 10:43:31: netdata DEBUG : PLUGIN[cgroups] : (1613@collectors/cgroups.p:read_cgroup ): reading metrics for cgroups '/lxc.payload.emailserver'
2022-12-15 10:43:31: netdata DEBUG : PLUGIN[cgroups] : (1613@collectors/cgroups.p:read_cgroup ): reading metrics for cgroups '/lxc.payload.reverseprox1'
2022-12-15 10:43:31: netdata DEBUG : PLUGIN[cgroups] : (1613@collectors/cgroups.p:read_cgroup ): reading metrics for cgroups '/lxc.payload.butterflyeu'
2022-12-15 10:43:31: netdata DEBUG : PLUGIN[cgroups] : (1613@collectors/cgroups.p:read_cgroup ): reading metrics for cgroups '/lxc.payload.peng'
2022-12-15 10:43:31: netdata DEBUG : PLUGIN[cgroups] : (1613@collectors/cgroups.p:read_cgroup ): reading metrics for cgroups '/lxc.payload.bipradix'
2022-12-15 10:43:31: netdata DEBUG : PLUGIN[cgroups] : (1613@collectors/cgroups.p:read_cgroup ): reading metrics for cgroups '/lxc.payload.ireallydo'
2022-12-15 10:43:31: netdata DEBUG : PLUGIN[cgroups] : (1613@collectors/cgroups.p:read_cgroup ): reading metrics for cgroups '/lxc.payload.ads'
2022-12-15 10:43:31: netdata DEBUG : PLUGIN[cgroups] : (1613@collectors/cgroups.p:read_cgroup ): reading metrics for cgroups '/lxc.payload.kindao'
2022-12-15 10:43:31: netdata DEBUG : PLUGIN[cgroups] : (1613@collectors/cgroups.p:read_cgroup ): reading metrics for cgroups '/lxc.payload.viecondocker'
2022-12-15 10:43:31: netdata DEBUG : PLUGIN[cgroups] : (1613@collectors/cgroups.p:read_cgroup ): reading metrics for cgroups '/lxc.payload.iran'
2022-12-15 10:43:31: netdata DEBUG : PLUGIN[cgroups] : (1613@collectors/cgroups.p:read_cgroup ): reading metrics for cgroups '/lxc.payload.aclserver'
2022-12-15 10:43:31: netdata DEBUG : PLUGIN[cgroups] : (1613@collectors/cgroups.p:read_cgroup ): reading metrics for cgroups '/lxc.payload.sapphirecap2'
2022-12-15 10:43:31: netdata DEBUG : PLUGIN[cgroups] : (1613@collectors/cgroups.p:read_cgroup ): reading metrics for cgroups '/lxc.payload.deloittefond'
2022-12-15 10:43:31: netdata DEBUG : PLUGIN[cgroups] : (1613@collectors/cgroups.p:read_cgroup ): reading metrics for cgroups '/lxc.payload.authservice'
2022-12-15 10:43:31: netdata DEBUG : PLUGIN[cgroups] : (1613@collectors/cgroups.p:read_cgroup ): reading metrics for cgroups '/lxc.payload.valtameri'
2022-12-15 10:43:31: netdata DEBUG : PLUGIN[cgroups] : (1613@collectors/cgroups.p:read_cgroup ): reading metrics for cgroups '/lxc.payload.darpa'
...
Downloads $ grep "disabled by" netdata_docker_logs.txt | grep -Eo "name '[^']+'" | sort | uniq | sed -e "s/^name '//" -e "s/'$//"
/
dev-hugepages.mount
dev-mqueue.mount
docker
lxc.monitor.aclserver
lxc.monitor.ads
lxc.monitor.adw
lxc.monitor.androphedia
lxc.monitor.authservice
lxc.monitor.bdcloudapps
lxc.monitor.bhtwebdock
lxc.monitor.bigcheese
lxc.monitor.bipradix
lxc.monitor.bnw
lxc.monitor.bot
lxc.monitor.butterflyeu
lxc.monitor.bytesahead
lxc.monitor.capitalcom
lxc.monitor.cardataapi
lxc.monitor.cdmain
lxc.monitor.chat
lxc.monitor.checkworf
lxc.monitor.cms
lxc.monitor.coreconnect
lxc.monitor.darpa
lxc.monitor.deloittefond
lxc.monitor.dhke
lxc.monitor.docker1
lxc.monitor.documotor
lxc.monitor.durarerp
lxc.monitor.dy11
lxc.monitor.emailserver
lxc.monitor.entegrali
lxc.monitor.escisnew
lxc.monitor.fourfpstudio
lxc.monitor.fynbusweb3
lxc.monitor.healthsy
lxc.monitor.hobbyserver
lxc.monitor.hordalan2023
lxc.monitor.hydrogen
lxc.monitor.india1
lxc.monitor.iran
lxc.monitor.ireallydo
lxc.monitor.istudyturkiy
lxc.monitor.jaczulloisno
lxc.monitor.jsq
lxc.monitor.justix
lxc.monitor.kindao
lxc.monitor.komeil1
lxc.monitor.lurianbase
lxc.monitor.maghalejoo2
lxc.monitor.maxthon
lxc.monitor.mc
lxc.monitor.microsrvc1
lxc.monitor.microsrvc2
lxc.monitor.microsrvc3
lxc.monitor.microsrvc4
lxc.monitor.mnd
lxc.monitor.moseacg
lxc.monitor.musrv
lxc.monitor.mynode6
lxc.monitor.mynode7
lxc.monitor.nako
lxc.monitor.nanodeck
lxc.monitor.ouwgredmine
lxc.monitor.oxen2
lxc.monitor.pbserver
lxc.monitor.peng
lxc.monitor.phs
lxc.monitor.planb
lxc.monitor.pre002finlan
lxc.monitor.projectzomb
lxc.monitor.puoti2022
lxc.monitor.randsomar
lxc.monitor.raspberry
lxc.monitor.redbullhel
lxc.monitor.reverseprox1
lxc.monitor.robertoarred
lxc.monitor.sapphirecap1
lxc.monitor.sapphirecap2
lxc.monitor.sapphirecap3
lxc.monitor.sapphirecap4
lxc.monitor.sapphirecapi
lxc.monitor.sapphirecapn
lxc.monitor.server6
lxc.monitor.shiftplanner
lxc.monitor.strapi
lxc.monitor.suissebankke
lxc.monitor.testrecovery
lxc.monitor.tntrr
lxc.monitor.ubuntu5
lxc.monitor.v2ui
lxc.monitor.valtameri
lxc.monitor.viecondocker
lxc.monitor.vldirsrv002
lxc.monitor.vm
lxc.monitor.vps210thread
lxc.monitor.vps410thread
lxc.monitor.vps55thread
lxc.monitor.weather
lxc.monitor.webdev
lxc.monitor.wienerslol
lxc.monitor.wp221967
lxc.monitor.wwsszz
lxc.pivot
proc-sys-fs-binfmt_misc.mount
sys-fs-fuse-connections.mount
sys-kernel-config.mount
sys-kernel-debug.mount
sys-kernel-tracing.mount
system.slice
system.slice/boot-efi.mount
system.slice/boot.mount
system.slice/dev-disk-by/x2did-dm/x2duuid/x2dlvm/x2di01qb1bwuulokcjaambdkxtzbizwa3euwk6uf32e4ey4bwndrlsbwzqortha8epe.swap
system.slice/docker.socket
system.slice/snap-core20-1405.mount
system.slice/snap-lxd-22923.mount
system.slice/snap-lxd-23680.mount
system.slice/snap-snapd-15534.mount
system.slice/snapd.socket
system.slice/system-getty.slice
system.slice/system-getty.slice/getty_tty1.service
system.slice/system-lvm2/x2dpvscan.slice
system.slice/system-modprobe.slice
system.slice/system-systemd/x2dfsck.slice
user.slice
Do you mean the Netdata docker container (netdata_test_cgroups) has no container metrics? (<IP>:20000)
Yes, after creating the container I visited the agent dashboard and I didn’t find the container metrics (usually I would find them in the bottom right)
I followed the instructions here and uncommented those lines and restarted Netdata, still container metrics are not shown.
I uncommented these lines (that “lxc.monitor” pattern line as well) and restarted Netdata but for some reason the container metrics are not shown.
Check the screenshot to see the changes I made.
Any input on this?
I think you need lxc.playload.* (no config changes are needed), not lxc.monitor.*. I don’t know what is happening. I see no problems in the logs. I will add more debug logs and build another custom image.
Is this correct? I am not sure what the right syntax is.
search for cgroups in subpaths matching = !*/init.scope !*-qemu !*.libvirt-qemu !/init.scope !/system !/systemd !/user !/user.slice !/lxc/*/* !/lxc.monitor !/lxc.payload/*/* /lxc.payload.* *
Sorry to bother but is that the right syntax? I guess an “!” means not to consider that, so I removed it and restarted Netdata but still nothing - no information on container metrics.
@ilyam8 Still the container metrics are not shown. Here is our current search pattern:
search for cgroups in subpaths matching = !*/init.scope !*-qemu !*.libvirt-qemu !/init.scope !/system !/systemd !/user !/user.slice !/lxc/*/* /lxc.monitor.* !/lxc.payload/*/* /lxc.payload.* *
In /sys/fs/cgroup/devices I could see the lxc containers (those “payload” and “monitor” thing) but nothing shows up in Netdata.
Not sure how to proceed. If needed we can provide you shell access to our server so that you can go in and see what’s wrong.
Thank you.
Any update on this? It would be very helpful and appreciated if you could look into this again.
I’m not sure how different these both are. I thought maybe this is some configuration issue of lxc or maybe the version of LXD hypervisor we’re using. But even docker containers are not visible in the Netdata dashboard. Not sure why. I am running the container under the root user.
Hi, @philip. I don’t know what the problem is. There is no problem according to the logs. I will try to add more logging and create another custom image ~ on Friday.