# Need some help with cgroups configuration for qemu VMs

**URL:** <https://community.netdata.cloud/t/need-some-help-with-cgroups-configuration-for-qemu-vms/1167>\
**Category:** Help\
**Tags:** agent-collector, agent\
**Created:** [April 12, 2021, 11:22am UTC](https://community.netdata.cloud/t/need-some-help-with-cgroups-configuration-for-qemu-vms/1167 "2021-04-12T11:22:51Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![ovi](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/ovi/32/686_2.png) [@ovi](https://community.netdata.cloud/u/ovi)\
**Post date:** [April 12, 2021, 11:22am UTC](https://community.netdata.cloud/t/need-some-help-with-cgroups-configuration-for-qemu-vms/1167/1 "2021-04-12T11:22:51Z")

</div>

Hi there,

I have installed netdata on a host where I run plenty of docker containers and out of the box I can see stats for each container:  
 ![image](https://canada1.discourse-cdn.com/flex029/uploads/netdata2/original/1X/b117bb9fe37b9081b64bec50ae656db1bc618e6a.png)

---

<div class="post-metadata">

**Author:** ![ovi](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/ovi/32/686_2.png) [@ovi](https://community.netdata.cloud/u/ovi)\
**Post date:** [April 12, 2021, 11:23am UTC](https://community.netdata.cloud/t/need-some-help-with-cgroups-configuration-for-qemu-vms/1167/2 "2021-04-12T11:23:24Z")

</div>

**As a new user I can only add one image per post ☹ hence my reply to my own topic.**

Next I installed netdata on a host where I run KVMs (using Proxmox qhich is based on qemu) but here I don’t see the stats per KVM but rather alltogether:  
 ![image](https://canada1.discourse-cdn.com/flex029/uploads/netdata2/original/1X/90fcc3985f7b650a4ff8a9871e5d597ced4c7ee8.png)

I found the instructions a bit beyond my understanding, maybe someone can help me configure netdata so that the resources used are grouped by VM?  
=\> [cgroups.plugin | Learn Netdata](https://learn.netdata.cloud/docs/agent/collectors/cgroups.plugin)

---

<div class="post-metadata">

**Author:** ![vlvkobal](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/vlvkobal/32/219_2.png) [@vlvkobal](https://community.netdata.cloud/u/vlvkobal)\
**Post date:** [April 12, 2021, 11:39am UTC](https://community.netdata.cloud/t/need-some-help-with-cgroups-configuration-for-qemu-vms/1167/3 "2021-04-12T11:39:53Z")

</div>

I don’t know if it will help but try to tweak

```auto
[plugin:cgroups]
	enable by default cgroups matching = !*/init.scope !/system.slice/run-*.scope *.scope /machine.slice/*.service /kubepods/pod*/* /kubepods/*/pod*/* !/kubepods* !*/vcpu* !*/emulator !*.mount !*.partition !*.service !*.socket !*.slice !*.swap !*.user !/ !/docker !/libvirt !/lxc !/lxc/*/* !/lxc.monitor* !/lxc.pivot !/lxc.payload !/machine !/qemu !/system !/systemd !/user * 
	search for cgroups in subpaths matching = !*/init.scope !*-qemu !*.libvirt-qemu !/init.scope !/system !/systemd !/user !/user.slice !/lxc/*/* !/lxc.monitor !/lxc.payload/*/* !/lxc.payload.* * 

```

in `netdata.conf`

---

<div class="post-metadata">

**Author:** ![OdysLam](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/odyslam/32/98_2.png) [@OdysLam](https://community.netdata.cloud/u/OdysLam)\
**Post date:** [April 12, 2021, 11:41am UTC](https://community.netdata.cloud/t/need-some-help-with-cgroups-configuration-for-qemu-vms/1167/4 "2021-04-12T11:41:51Z")

</div>

Thanks @vlvkobal, this is the answer. Try to remove the `!*-qemu` bit in the configuration and then [restart Netdata](https://learn.netdata.cloud/docs/configure/start-stop-restart).

---

<div class="post-metadata">

**Author:** ![ilyam8](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/ilyam8/32/134_2.png) [@ilyam8](https://community.netdata.cloud/u/ilyam8)\
**Post date:** [April 12, 2021, 12:31pm UTC](https://community.netdata.cloud/t/need-some-help-with-cgroups-configuration-for-qemu-vms/1167/5 "2021-04-12T12:31:55Z")

</div>

> but here I don’t see the stats per KVM but rather alltogether

@vlvkobal your fix filters out kvm all qemu stats, doesn’t it?

I found the issue [QEMU metrics incorrect and crashing dashboard · Issue #9254 · netdata/netdata · GitHub](https://github.com/netdata/netdata/issues/9254)

---

<div class="post-metadata">

**Author:** ![ovi](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/ovi/32/686_2.png) [@ovi](https://community.netdata.cloud/u/ovi)\
**Post date:** [April 12, 2021, 12:35pm UTC](https://community.netdata.cloud/t/need-some-help-with-cgroups-configuration-for-qemu-vms/1167/6 "2021-04-12T12:35:53Z")

</div>

Thanks everyone. I removed the “!\*-qemu” bit in both lines (enable by default cgroups matching = and search for cgroups in subpaths matching =) and restarted netdata.

I am not accessing netdata locally - only through netdata cloud. A few refreshes later I still see:  
 ![image](https://canada1.discourse-cdn.com/flex029/uploads/netdata2/original/1X/ace7a6b3121221733e3606d6d47f602878b94851.png)

---

<div class="post-metadata">

**Author:** ![vlvkobal](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/vlvkobal/32/219_2.png) [@vlvkobal](https://community.netdata.cloud/u/vlvkobal)\
**Post date:** [April 12, 2021, 12:42pm UTC](https://community.netdata.cloud/t/need-some-help-with-cgroups-configuration-for-qemu-vms/1167/7 "2021-04-12T12:42:08Z")

</div>

> [@ilyam8](#):
>
> @vlvkobal your fix filters out kvm all qemu stats, doesn’t it?

That isn’t a fix, it’s just our default config. I don’t know what is the structure of containers in QEMU, that’s why I suggested tweaking the configuration parameters, rather than providing a more specific suggestion.

---

<div class="post-metadata">

**Author:** ![ovi](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/ovi/32/686_2.png) [@ovi](https://community.netdata.cloud/u/ovi)\
**Post date:** [April 12, 2021, 12:45pm UTC](https://community.netdata.cloud/t/need-some-help-with-cgroups-configuration-for-qemu-vms/1167/8 "2021-04-12T12:45:31Z")

</div>

Thanks for your pointer, I understand. I had hoped someone would be using qemu and netdata and might be able to provide more help.

Lets give it a few days, maybe someone will spot this thread 🙂

---

<div class="post-metadata">

**Author:** ![ilyam8](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/ilyam8/32/134_2.png) [@ilyam8](https://community.netdata.cloud/u/ilyam8)\
**Post date:** [April 12, 2021, 12:52pm UTC](https://community.netdata.cloud/t/need-some-help-with-cgroups-configuration-for-qemu-vms/1167/9 "2021-04-12T12:52:02Z")

</div>

That is cgroup directory structure on my server

```nohighlight
[ilyam@pc ~]$ tree -d -L 5 /sys/fs/cgroup/cpu/
/sys/fs/cgroup/cpu/
├── dev-hugepages.mount
├── dev-mqueue.mount
├── docker
│ └── d9484ad246a254c96d52f9e3a0b414a80026bf5a17c560cc648d640cd9e98793
├── init.scope
├── -.mount
├── proc-sys-fs-binfmt_misc.mount
├── sys-fs-fuse-connections.mount
├── sys-kernel-config.mount
├── sys-kernel-debug.mount
├── sys-kernel-tracing.mount
├── system.slice
│ ├── apparmor.service
│ ├── avahi-daemon.service
│ ├── avahi-daemon.socket
│ ├── boot-efi.mount
│ ├── cronie.service
│ ├── dbus.service
│ ├── dbus.socket
│ ├── dm-event.socket
│ ├── docker.service
│ ├── docker.socket
│ ├── kmod-static-nodes.service
│ ├── lm_sensors.service
│ ├── lvm2-lvmpolld.socket
│ ├── lvm2-monitor.service
│ ├── ModemManager.service
│ ├── NetworkManager.service
│ ├── NetworkManager-wait-online.service
│ ├── ntpd.service
│ ├── polkit.service
│ ├── rtkit-daemon.service
│ ├── run-user-1000-gvfs.mount
│ ├── run-user-1000.mount
│ ├── sddm.service
│ ├── smartd.service
│ ├── snapd.apparmor.service
│ ├── snapd.socket
│ ├── sshd.service
│ ├── systemd-binfmt.service
│ ├── systemd-coredump.socket
│ ├── systemd-journald-audit.socket
│ ├── systemd-journald-dev-log.socket
│ ├── systemd-journald.service
│ ├── systemd-journald.socket
│ ├── systemd-journal-flush.service
│ ├── systemd-logind.service
│ ├── systemd-modules-load.service
│ ├── systemd-random-seed.service
│ ├── systemd-remount-fs.service
│ ├── systemd-rfkill.socket
│ ├── systemd-sysctl.service
│ ├── systemd-tmpfiles-setup-dev.service
│ ├── systemd-tmpfiles-setup.service
│ ├── systemd-udevd-control.socket
│ ├── systemd-udevd-kernel.socket
│ ├── systemd-udevd.service
│ ├── systemd-udev-trigger.service
│ ├── systemd-update-utmp.service
│ ├── systemd-user-sessions.service
│ ├── system-getty.slice
│ ├── system-modprobe.slice
│ ├── system-systemd\x2dcoredump.slice
│ ├── system-systemd\x2dfsck.slice
│ ├── tlp.service
│ ├── tmp.mount
│ ├── udisks2.service
│ ├── upower.service
│ └── wpa_supplicant.service
└── user.slice

70 directories

```

I have one docker container running and we can see it

```auto
├── docker
│ └── d9484ad246a254c96d52f9e3a0b414a80026bf5a17c560cc648d640cd9e98793

```

* * *

@ovi lets see what you have

---

<div class="post-metadata">

**Author:** ![ovi](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/ovi/32/686_2.png) [@ovi](https://community.netdata.cloud/u/ovi)\
**Post date:** [April 12, 2021, 1:05pm UTC](https://community.netdata.cloud/t/need-some-help-with-cgroups-configuration-for-qemu-vms/1167/10 "2021-04-12T13:05:57Z")

</div>

> [@ilyam8](#):
>
> `tree -d -L 5 /sys/fs/cgroup/cpu/`

Thanks!  
I cut the output of that command as the qemu part is at the top, that should give you enough info or do you need the rest?

```
tree -d -L 5 /sys/fs/cgroup/cpu/

/sys/fs/cgroup/cpu/
├── init.scope
├── qemu.slice
│ ├── 100.scope
│ ├── 101.scope
│ └── 102.scope
├── system.slice
│ ├── apparmor.service
│ ├── blk-availability.service
│ ├── console-setup.service
│ ├── cron.service

```

---

<div class="post-metadata">

**Author:** ![ovi](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/ovi/32/686_2.png) [@ovi](https://community.netdata.cloud/u/ovi)\
**Post date:** [April 12, 2021, 1:08pm UTC](https://community.netdata.cloud/t/need-some-help-with-cgroups-configuration-for-qemu-vms/1167/11 "2021-04-12T13:08:54Z")

</div>

Just to clarify, I expected to see stats for the 3 VMs separately grouped underneath the qemu menu in netdata, just like it is show for each docker container.  
(Just making sure I properly explained things in my first to posts)

This is one sample container:  
 ![image](https://canada1.discourse-cdn.com/flex029/uploads/netdata2/original/1X/dbc10a136c883777a8469a9ae87d09a2bbec2430.png)

---

<div class="post-metadata">

**Author:** ![ovi](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/ovi/32/686_2.png) [@ovi](https://community.netdata.cloud/u/ovi)\
**Post date:** [April 12, 2021, 1:10pm UTC](https://community.netdata.cloud/t/need-some-help-with-cgroups-configuration-for-qemu-vms/1167/12 "2021-04-12T13:10:40Z")

</div>

qemu are all together, not grouped by “slice”:  
 ![image](https://canada1.discourse-cdn.com/flex029/uploads/netdata2/original/1X/513f736789e077378371a4ec2c35e82d651b082b.png)

---

<div class="post-metadata">

**Author:** ![ilyam8](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/ilyam8/32/134_2.png) [@ilyam8](https://community.netdata.cloud/u/ilyam8)\
**Post date:** [April 12, 2021, 1:22pm UTC](https://community.netdata.cloud/t/need-some-help-with-cgroups-configuration-for-qemu-vms/1167/13 "2021-04-12T13:22:03Z")

</div>

@ovi

Show `ls -l /etc/pve/qemu-server/`

We expect to see `/etc/pve/qemu-server/100.conf`(101, 102) in there. And we need `/etc/pve/qemu-server/100.conf` content to identify the problem, please share.

---

<div class="post-metadata">

**Author:** ![ovi](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/ovi/32/686_2.png) [@ovi](https://community.netdata.cloud/u/ovi)\
**Post date:** [April 12, 2021, 1:25pm UTC](https://community.netdata.cloud/t/need-some-help-with-cgroups-configuration-for-qemu-vms/1167/14 "2021-04-12T13:25:11Z")

</div>

> [@ilyam8](#):
>
> ls -l /etc/pve/qemu-server/

```
ls -l /etc/pve/qemu-server/

total 2
-rw-r----- 1 root www-data 396 Apr 12 05:51 100.conf
-rw-r----- 1 root www-data 1012 Apr 12 08:50 101.conf
-rw-r----- 1 root www-data 382 Apr 12 13:11 102.conf

 cat /etc/pve/qemu-server/100.conf

#Ubuntu 20.4
#mailcow
agent: 1
boot: order=ide2;scsi0;net0
cores: 2
ide2: none,media=cdrom
memory: 9216
name: mail
net0: virtio=CA:18:5B:74:78:78,bridge=vmbr0
numa: 0
onboot: 1
ostype: l26
scsi0: local-zfs:vm-100-disk-0,discard=on,size=128G
scsihw: virtio-scsi-pci
smbios1: uuid=34fdaae9-f689-4903-8f17-f683578ee6bb
sockets: 1
startup: up=60,down=60
vmgenid: f8209b9e-fc9e-4a2f-9c51-bb5995a2c5fb

```

---

<div class="post-metadata">

**Author:** ![ilyam8](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/ilyam8/32/134_2.png) [@ilyam8](https://community.netdata.cloud/u/ilyam8)\
**Post date:** [April 12, 2021, 1:27pm UTC](https://community.netdata.cloud/t/need-some-help-with-cgroups-configuration-for-qemu-vms/1167/15 "2021-04-12T13:27:49Z")

</div>

We suspect that the problem is cgroups name resolution script.

A quick fix is to disable name resolution for qemu cgroups

add this line to the netdata.conf to the `[plugin:cgroups]` section

```auto
run script to rename cgroups matching = !/ !*.mount !*.socket !*.partition /machine.slice/*.service !*.service !*.slice !*.swap !*.user !init.scope !*.scope/vcpu* !*.scope/emulator *.scope *docker* *lxc* !*qemu* *kubepods* *.libvirt-qemu *

```

Don’t forget to restart netdata service after the changes.

---

<div class="post-metadata">

**Author:** ![ovi](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/ovi/32/686_2.png) [@ovi](https://community.netdata.cloud/u/ovi)\
**Post date:** [April 12, 2021, 1:38pm UTC](https://community.netdata.cloud/t/need-some-help-with-cgroups-configuration-for-qemu-vms/1167/16 "2021-04-12T13:38:30Z")

</div>

in my /etc/netdata/netdata.conf I had that line already but commented. I un-commented it and compared to yours - they are identical but there are still no changes.  
 ![image](https://canada1.discourse-cdn.com/flex029/uploads/netdata2/original/1X/1900583aa08e9700d2edb162098e614db0cd35d0.png)

Just to summarize:

In these two lines I have removed the “!\*-qemu” bits

```
enable by default cgroups matching
search for cgroups in subpaths matching

```

while this line:  
`run script to rename cgroups matching`

still contains it.

---

<div class="post-metadata">

**Author:** ![ilyam8](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/ilyam8/32/134_2.png) [@ilyam8](https://community.netdata.cloud/u/ilyam8)\
**Post date:** [April 12, 2021, 1:40pm UTC](https://community.netdata.cloud/t/need-some-help-with-cgroups-configuration-for-qemu-vms/1167/17 "2021-04-12T13:40:12Z")

</div>

default `run script to rename cgroups matching` contains `*qemu*`, i changed it to `!*qemu*`

---

<div class="post-metadata">

**Author:** ![ovi](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/ovi/32/686_2.png) [@ovi](https://community.netdata.cloud/u/ovi)\
**Post date:** [April 12, 2021, 1:44pm UTC](https://community.netdata.cloud/t/need-some-help-with-cgroups-configuration-for-qemu-vms/1167/18 "2021-04-12T13:44:56Z")

</div>

sorry, my mistake thanks for spotting it but even with that change I see no result ☹

---

<div class="post-metadata">

**Author:** ![ilyam8](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/ilyam8/32/134_2.png) [@ilyam8](https://community.netdata.cloud/u/ilyam8)\
**Post date:** [April 12, 2021, 1:45pm UTC](https://community.netdata.cloud/t/need-some-help-with-cgroups-configuration-for-qemu-vms/1167/19 "2021-04-12T13:45:38Z")

</div>

Just to be sure, you’ve restarted netdata servie after the changes?

* * *

Anyway, let’s check logs

> grep -i cgroup-name error.log

---

<div class="post-metadata">

**Author:** ![ovi](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/ovi/32/686_2.png) [@ovi](https://community.netdata.cloud/u/ovi)\
**Post date:** [April 12, 2021, 1:54pm UTC](https://community.netdata.cloud/t/need-some-help-with-cgroups-configuration-for-qemu-vms/1167/20 "2021-04-12T13:54:33Z")

</div>

> [@ilyam8](#):
>
> > grep -i cgroup-name error.log

Yes I did, after every change.

Thanks for the pointer with the netdata logs, I guess now all is clear:

```
/var/log/netdata# grep -i cgroup-name error.log
2021-04-12 13:05:24: cgroup-name.sh: ERROR: proxmox config file missing /etc/pve/qemu-server/100.conf or netdata does not have read access. Please ensure netdata is a member of www-data group.

```

fixed by adding the user to the group:  
`adduser netdata www-data`

check:  
groups netdata  
netdata : netdata adm proxy www-data ceph

restarted netdata and it works:

![image](https://canada1.discourse-cdn.com/flex029/uploads/netdata2/original/1X/398de9bee91678da11175ff92758f7849052bbe8.png)

Do you reckon I still need all those edits we made during this thread except for adding netdata to the www-data group?

[Next page](https://community.netdata.cloud/t/need-some-help-with-cgroups-configuration-for-qemu-vms/1167.md?page=2)