Cannot read disk space usage metrics && wrong metrics inside Docker/Kubernetes environment (important because we should alert when disks become full)

fzyzcjy · February 8, 2021, 12:22pm

Bug report summary

Hi thanks for the lib! I find the “disk” section incomplete. It does not have disk_spaces._/disk_usages._ chart for most of the disks (e.g. sda, sdb, … and dm-1, dm-2, … It only has that for a few disks like /.

In addition, even for the / filesystem, the disk size that netdata says is different from the actual value. When execute df -h in my host machine, it says /dev/vda1 40G 3.5G 34G 10% /. But as you can see from figure 1, netdata think it is 8GB (not 40GB).

I guess it is related to permission problems and problems of container volume mapping? Possibly related (not sure): Disk space not available · Issue #3716 · netdata/netdata · GitHub. But I do not have any ideas on how to solve this.

Thanks!

figure 1: it has metrics for /

figure 2: it does not for sda

OS / Environment

kubernetes on centos 7

Netdata version

latest

Component Name

disk

Steps To Reproduce

install official helmchart
see it

Expected behavior

should see correct metrics

I have already tried to mount /mnt to /host/mnt (hoping that things like /mnt/sda, /mnt/sdb, etc can be found), but no success. Nothing more is appeared in the UI. Any suggestions?

vlvkobal · February 8, 2021, 12:55pm

What are /mnt/sda and /mnt/sdb ? Are they mounted filesystems?

They are just normal disk partitions(?). You know, just those paths seen everywhere when you have a linux machine.

Do you see them when you run df inside the container?

vlvkobal · February 8, 2021, 1:00pm

Oh, wait, they are not filesystems, right? Then you can’t see them. Only mounted filesystems are shown by the diskspace plugin.

fzyzcjy · February 8, 2021, 1:33pm

@vlvkobal Sorry I am new to Linux, so ignore my “they are just normal disk partitions”. I did not clearly used those terms.

Here are df reults:

In the “raw” host machine, sudo df -h

/dev/vda1                       40G  3.6G   34G   10% /
devtmpfs                       7.8G     0  7.8G    0% /dev
tmpfs                          7.8G     0  7.8G    0% /dev/shm
tmpfs                          7.8G  725M  7.1G   10% /run
tmpfs                          7.8G     0  7.8G    0% /sys/fs/cgroup
/dev/mapper/vgpaas-kubernetes  9.8G  348M  9.0G    4% /mnt/paas/kubernetes/kubelet
/dev/mapper/vgpaas-dockersys    18G  2.0G   15G   12% /var/lib/docker
tmpfs                          7.8G   12K  7.8G    1% /mnt/paas/kubernetes/kubelet/pods/3f849fbd-d798-40eb-9f55-76f72c41e7ed/volumes/kubernetes.io~secret/everest-csi-plugin-token-h7fkd
tmpfs                          7.8G   12K  7.8G    1% /mnt/paas/kubernetes/kubelet/pods/4ab028cc-a217-46f4-acd3-e1ea4191c0ad/volumes/kubernetes.io~secret/default-token-9cqqb
tmpfs                          7.8G   12K  7.8G    1% /mnt/paas/kubernetes/kubelet/pods/9e1c7699-9f65-480f-8f57-611d495c6906/volumes/kubernetes.io~secret/everest-csi-plugin-token-h7fkd
... and many tmpfs...
/kubernetes.io~secret/cceaddon-npd-token-p4rx4
/dev/dm-5                      9.8G   37M  9.3G    1% /var/lib/docker/devicemapper/mnt/2ae2a1b32d875f8537e43057acb3b453f59501382ddc25374e7af6ad2327ec89
/dev/dm-6                      9.8G   37M  9.3G    1% /var/lib/docker/devicemapper/mnt/01166ef173b156bc4120f233ca64eba9f9d1ab6874033d2e9e8cdaceb88cb9cb
shm                             64M     0   64M    0% /var/lib/docker/containers/6543af33e179ffb497c7ab52ea0d0c14ccc0bc734532a3d102299b69f4b02d80/mounts/shm
/dev/dm-11                     9.8G   37M  9.3G    1% /var/lib/docker/devicemapper/mnt/59426f521125566af9a61e3167d62e83abef0e9a8f608b17c8d2c3fb73e0bbd7
shm                             64M     0   64M    0% /var/lib/docker/containers/4f826d7847a4b7ab7b9d7f4f2e4cfa71cbed3392e58a8009e902f6e028dc72a8/mounts/shm
/dev/dm-9                      9.8G   37M  9.3G    1% /var/lib/docker/devicemapper/mnt/bf8ad325c5cb8e32b08abc5da41986807ae6899c96467268d2b2e43c6b7dfb5f
... and many dm...
/dev/sdb                       7.9G  881M  7.0G   12% /mnt/paas/kubernetes/kubelet/plugins/kubernetes.io/csi/pv/pvc-c6bd0461-dfd0-4c94-953a-5d79eaa9e523/globalmount
/dev/sda                       7.9G  1.1G  6.8G   14% /mnt/paas/kubernetes/kubelet/plugins/kubernetes.io/csi/pv/pvc-51a8253b-76f0-4825-9348-17650e464f42/globalmount
/dev/sdc                       7.9G  1.2G  6.7G   15% /mnt/paas/kubernetes/kubelet/plugins/kubernetes.io/csi/pv/pvc-af3a108c-2f6c-4794-b09e-6bff71b93e57/globalmount
... and many sd...

Here I enter netdata child container (of netdata-child pod of kubernetes) and run df -h

bash-5.0# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/mapper/docker-252:1-393217-d70f82f344818e7965967c00d3ebf2a646da9709925ba1ec5c7e4e75d5c62a3e
                          9.8G    316.9M      9.0G   3% /
tmpfs                    64.0M         0     64.0M   0% /dev
tmpfs                     7.7G         0      7.7G   0% /sys/fs/cgroup
/dev/mapper/vgpaas-kubernetes
                          9.8G    347.3M      8.9G   4% /dev/termination-log
/dev/vda1                39.1G      3.5G     33.6G   9% /host/mnt
/dev/mapper/vgpaas-kubernetes
                          9.8G    347.3M      8.9G   4% /host/mnt/paas/kubernetes/kubelet
tmpfs                     7.7G     12.0K      7.7G   0% /host/mnt/paas/kubernetes/kubelet/pods/3f849fbd-d798-40eb-9f55-76f72c41e7ed/volumes/kubernetes.io~secret/everest-csi-plugin-token-h7fkd
tmpfs                     7.7G     12.0K      7.7G   0% /host/mnt/paas/kubernetes/kubelet/pods/4ab028cc-a217-46f4-acd3-e1ea4191c0ad/volumes/kubernetes.io~secret/default-token-9cqqb
tmpfs                     7.7G     12.0K      7.7G   0% /host/mnt/paas/kubernetes/kubelet/pods/9e1c7699-9f65-480f-8f57-611d495c6906/volumes/kubernetes.io~secret/everest-csi-plugin-token-h7fkd
tmpfs                     7.7G     12.0K      7.7G   0% /host/mnt/paas/kubernetes/kubelet/pods/a9f9f061-e9ec-4aa1-8579-d0e1269d18a8/volumes/kubernetes.io~secret/default-token-gbb8r
tmpfs                     7.7G     12.0K      7.7G   0% /host/mnt/paas/kubernetes/kubelet/pods/7446443d-dcfc-4b98-a3eb-a9b133e192f8/volumes/kubernetes.io~secret/coredns-token-gc7kn
...
/dev/sdb                  7.8G    881.0M      6.9G  11% /host/mnt/paas/kubernetes/kubelet/plugins/kubernetes.io/csi/pv/pvc-c6bd0461-dfd0-4c94-953a-5d79eaa9e523/globalmount
/dev/sdb                  7.8G    881.0M      6.9G  11% /host/mnt/paas/kubernetes/kubelet/pods/7920b528-6201-45b6-8f87-15abce67c627/volumes/kubernetes.io~csi/pvc-c6bd0461-dfd0-4c94-953a-5d79eaa9e523/mount
/dev/sda                  7.8G      1.1G      6.7G  14% /host/mnt/paas/kubernetes/kubelet/plugins/kubernetes.io/csi/pv/pvc-51a8253b-76f0-4825-9348-17650e464f42/globalmount
/dev/sda                  7.8G      1.1G      6.7G  14% /host/mnt/paas/kubernetes/kubelet/pods/89003534-ca01-4c1e-afe3-8bcfda2165bd/volumes/kubernetes.io~csi/pvc-51a8253b-76f0-4825-9348-17650e464f42/mount
/dev/sdc                  7.8G      1.2G      6.6G  15% /host/mnt/paas/kubernetes/kubelet/plugins/kubernetes.io/csi/pv/pvc-af3a108c-2f6c-4794-b09e-6bff71b93e57/globalmount
/dev/sdc                  7.8G      1.2G      6.6G  15% /host/mnt/paas/kubernetes/kubelet/pods/2f02ee9a-4479-4371-94f7-e5ea4269cc44/volumes/kubernetes.io~csi/pvc-af3a108c-2f6c-4794-b09e-6bff71b93e57/mount
...

(but do not see any dm- things - but that is ok)

fzyzcjy · February 8, 2021, 1:34pm

@vlvkobal

Restate my goal: I need to monitor the Persistent Volumes of kubernetes. For example, pod A may require a persistent volume of 40GB. Then I want to get an alert when it is going to use up all the space, such that I can take action by increasing the disk size.

Thanks very much!

vlvkobal · February 8, 2021, 3:55pm

Don’t you have /host/mnt/paas/kubernetes/kubelet/pods/* in the Netdata dashboard? Please make a new snapshot of your Disk section.

fzyzcjy · February 8, 2021, 11:43pm

@vlvkobal Thanks!

I do have things like k8s cntr default mypodname. But they do not show disk utilization.

fzyzcjy · February 8, 2021, 11:44pm

P.S. More info: my pod spec yaml (visualized in Lens)

fzyzcjy · February 9, 2021, 5:58am

By the way I try to debug the diskspace.plugin like what i did for go.d.plugin, but do not see any executable like go.d.plugin. (Seems that it is embedded)

fzyzcjy · February 9, 2021, 6:02am

OK I see logs. Let me try to change permission

2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet' (disk '_host_mnt_paas_kubernetes_kubelet', filesystem 'ext4', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/3f849fbd-d798-40eb-9f55-76f72c41e7ed/volumes/kubernetes.io~secret/everest-csi-plugin-token-h7fkd' (disk '_host_mnt_paas_kubernetes_kubelet_pods_3f849fbd-d798-40eb-9f55-76f72c41e7ed_volumes_kubernetes.io_secret_everest-csi-plugin-token-h7fkd', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/4ab028cc-a217-46f4-acd3-e1ea4191c0ad/volumes/kubernetes.io~secret/default-token-9cqqb' (disk '_host_mnt_paas_kubernetes_kubelet_pods_4ab028cc-a217-46f4-acd3-e1ea4191c0ad_volumes_kubernetes.io_secret_default-token-9cqqb', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/9e1c7699-9f65-480f-8f57-611d495c6906/volumes/kubernetes.io~secret/everest-csi-plugin-token-h7fkd' (disk '_host_mnt_paas_kubernetes_kubelet_pods_9e1c7699-9f65-480f-8f57-611d495c6906_volumes_kubernetes.io_secret_everest-csi-plugin-token-h7fkd', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/a9f9f061-e9ec-4aa1-8579-d0e1269d18a8/volumes/kubernetes.io~secret/default-token-gbb8r' (disk '_host_mnt_paas_kubernetes_kubelet_pods_a9f9f061-e9ec-4aa1-8579-d0e1269d18a8_volumes_kubernetes.io_secret_default-token-gbb8r', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/7446443d-dcfc-4b98-a3eb-a9b133e192f8/volumes/kubernetes.io~secret/coredns-token-gc7kn' (disk '_host_mnt_paas_kubernetes_kubelet_pods_7446443d-dcfc-4b98-a3eb-a9b133e192f8_volumes_kubernetes.io_secret_coredns-token-gc7kn', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/4346f106-c33a-4c70-b21c-f5f75ce2b37a/volumes/kubernetes.io~secret/strimzi-cluster-operator-token-jhgws' (disk '_host_mnt_paas_kubernetes_kubelet_pods_4346f106-c33a-4c70-b21c-f5f75ce2b37a_volumes_kubernetes.io_secret_strimzi-cluster-operator-token-jhgws', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/d888c0df-d78b-48c6-918d-4088b4bc0d0d/volumes/kubernetes.io~secret/default-token-9cqqb' (disk '_host_mnt_paas_kubernetes_kubelet_pods_d888c0df-d78b-48c6-918d-4088b4bc0d0d_volumes_kubernetes.io_secret_default-token-9cqqb', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/6f344c2e-2316-4b94-be28-804579fcf3fa/volumes/kubernetes.io~secret/default-token-9cqqb' (disk '_host_mnt_paas_kubernetes_kubelet_pods_6f344c2e-2316-4b94-be28-804579fcf3fa_volumes_kubernetes.io_secret_default-token-9cqqb', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/78a3a730-08dc-4ad7-a95f-ff02cb445222/volumes/kubernetes.io~secret/default-token-9cqqb' (disk '_host_mnt_paas_kubernetes_kubelet_pods_78a3a730-08dc-4ad7-a95f-ff02cb445222_volumes_kubernetes.io_secret_default-token-9cqqb', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/c015128a-2292-43ad-a6bc-00e6427b67c1/volumes/kubernetes.io~secret/default-token-9cqqb' (disk '_host_mnt_paas_kubernetes_kubelet_pods_c015128a-2292-43ad-a6bc-00e6427b67c1_volumes_kubernetes.io_secret_default-token-9cqqb', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/a9959c5f-da06-4158-861e-b069fffe68bd/volumes/kubernetes.io~secret/metrics-server-token-7tq9k' (disk '_host_mnt_paas_kubernetes_kubelet_pods_a9959c5f-da06-4158-861e-b069fffe68bd_volumes_kubernetes.io_secret_metrics-server-token-7tq9k', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/1896986d-2e92-4df4-8517-4855076aee71/volumes/kubernetes.io~secret/elastic-operator-token-tzsjd' (disk '_host_mnt_paas_kubernetes_kubelet_pods_1896986d-2e92-4df4-8517-4855076aee71_volumes_kubernetes.io_secret_elastic-operator-token-tzsjd', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/1896986d-2e92-4df4-8517-4855076aee71/volumes/kubernetes.io~secret/cert' (disk '_host_mnt_paas_kubernetes_kubelet_pods_1896986d-2e92-4df4-8517-4855076aee71_volumes_kubernetes.io_secret_cert', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/27e60571-8aaa-4d26-b11d-c802c62105d8/volumes/kubernetes.io~secret/cceaddon-npd-token-p4rx4' (disk '_host_mnt_paas_kubernetes_kubelet_pods_27e60571-8aaa-4d26-b11d-c802c62105d8_volumes_kubernetes.io_secret_cceaddon-npd-token-p4rx4', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/28e9bc77-8922-4c08-b5bf-4529b1a6ef40/volumes/kubernetes.io~secret/default-token-9cqqb' (disk '_host_mnt_paas_kubernetes_kubelet_pods_28e9bc77-8922-4c08-b5bf-4529b1a6ef40_volumes_kubernetes.io_secret_default-token-9cqqb', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/8ffd7eba-c20a-4a8e-b4a7-0331daa74cce/volumes/kubernetes.io~secret/tom-kafka-entity-operator-token-ldh6b' (disk '_host_mnt_paas_kubernetes_kubelet_pods_8ffd7eba-c20a-4a8e-b4a7-0331daa74cce_volumes_kubernetes.io_secret_tom-kafka-entity-operator-token-ldh6b', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/8ffd7eba-c20a-4a8e-b4a7-0331daa74cce/volumes/kubernetes.io~secret/cluster-ca-certs' (disk '_host_mnt_paas_kubernetes_kubelet_pods_8ffd7eba-c20a-4a8e-b4a7-0331daa74cce_volumes_kubernetes.io_secret_cluster-ca-certs', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/8ffd7eba-c20a-4a8e-b4a7-0331daa74cce/volumes/kubernetes.io~secret/eo-certs' (disk '_host_mnt_paas_kubernetes_kubelet_pods_8ffd7eba-c20a-4a8e-b4a7-0331daa74cce_volumes_kubernetes.io_secret_eo-certs', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/f5fd30e9-5d80-4fba-8d45-885ff14f9232/volumes/kubernetes.io~secret/kube-state-metrics-token-l6ccs' (disk '_host_mnt_paas_kubernetes_kubelet_pods_f5fd30e9-5d80-4fba-8d45-885ff14f9232_volumes_kubernetes.io_secret_kube-state-metrics-token-l6ccs', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/07c63238-5a28-413b-be86-9a316765013a/volumes/kubernetes.io~secret/static-linker-token-jf42q' (disk '_host_mnt_paas_kubernetes_kubelet_pods_07c63238-5a28-413b-be86-9a316765013a_volumes_kubernetes.io_secret_static-linker-token-jf42q', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/5ac7d853-813f-431c-bb5e-7d536fcc6dee/volumes/kubernetes.io~secret/static-linker-token-jf42q' (disk '_host_mnt_paas_kubernetes_kubelet_pods_5ac7d853-813f-431c-bb5e-7d536fcc6dee_volumes_kubernetes.io_secret_static-linker-token-jf42q', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/6923d9f5-add0-4cbd-8238-c8832fb197a3/volumes/kubernetes.io~secret/elastic-internal-kibana-config' (disk '_host_mnt_paas_kubernetes_kubelet_pods_6923d9f5-add0-4cbd-8238-c8832fb197a3_volumes_kubernetes.io_secret_elastic-internal-kibana-config', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/6923d9f5-add0-4cbd-8238-c8832fb197a3/volumes/kubernetes.io~secret/elasticsearch-certs' (disk '_host_mnt_paas_kubernetes_kubelet_pods_6923d9f5-add0-4cbd-8238-c8832fb197a3_volumes_kubernetes.io_secret_elasticsearch-certs', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/6923d9f5-add0-4cbd-8238-c8832fb197a3/volumes/kubernetes.io~secret/elastic-internal-http-certificates' (disk '_host_mnt_paas_kubernetes_kubelet_pods_6923d9f5-add0-4cbd-8238-c8832fb197a3_volumes_kubernetes.io_secret_elastic-internal-http-certificates', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/af259333-0c75-4ec7-b313-09173928b233/volumes/kubernetes.io~secret/default-token-9cqqb' (disk '_host_mnt_paas_kubernetes_kubelet_pods_af259333-0c75-4ec7-b313-09173928b233_volumes_kubernetes.io_secret_default-token-9cqqb', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/f7187e82-f8f3-411e-b2d5-135c2d05098b/volumes/kubernetes.io~secret/config' (disk '_host_mnt_paas_kubernetes_kubelet_pods_f7187e82-f8f3-411e-b2d5-135c2d05098b_volumes_kubernetes.io_secret_config', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/f7187e82-f8f3-411e-b2d5-135c2d05098b/volumes/kubernetes.io~secret/elasticsearch-certs' (disk '_host_mnt_paas_kubernetes_kubelet_pods_f7187e82-f8f3-411e-b2d5-135c2d05098b_volumes_kubernetes.io_secret_elasticsearch-certs', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/f7187e82-f8f3-411e-b2d5-135c2d05098b/volumes/kubernetes.io~secret/elastic-internal-http-certificates' (disk '_host_mnt_paas_kubernetes_kubelet_pods_f7187e82-f8f3-411e-b2d5-135c2d05098b_volumes_kubernetes.io_secret_elastic-internal-http-certificates', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/de7c0cc1-db35-48bf-a0b9-cb57d686e056/volumes/kubernetes.io~secret/kibana-certs' (disk '_host_mnt_paas_kubernetes_kubelet_pods_de7c0cc1-db35-48bf-a0b9-cb57d686e056_volumes_kubernetes.io_secret_kibana-certs', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/de7c0cc1-db35-48bf-a0b9-cb57d686e056/volumes/kubernetes.io~secret/elasticsearch-certs' (disk '_host_mnt_paas_kubernetes_kubelet_pods_de7c0cc1-db35-48bf-a0b9-cb57d686e056_volumes_kubernetes.io_secret_elasticsearch-certs', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/de7c0cc1-db35-48bf-a0b9-cb57d686e056/volumes/kubernetes.io~secret/config' (disk '_host_mnt_paas_kubernetes_kubelet_pods_de7c0cc1-db35-48bf-a0b9-cb57d686e056_volumes_kubernetes.io_secret_config', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/8df79e4a-587e-495e-9f97-54befabafc6a/volumes/kubernetes.io~secret/config' (disk '_host_mnt_paas_kubernetes_kubelet_pods_8df79e4a-587e-495e-9f97-54befabafc6a_volumes_kubernetes.io_secret_config', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/8df79e4a-587e-495e-9f97-54befabafc6a/volumes/kubernetes.io~secret/kibana-certs' (disk '_host_mnt_paas_kubernetes_kubelet_pods_8df79e4a-587e-495e-9f97-54befabafc6a_volumes_kubernetes.io_secret_kibana-certs', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/8df79e4a-587e-495e-9f97-54befabafc6a/volumes/kubernetes.io~secret/elasticsearch-certs' (disk '_host_mnt_paas_kubernetes_kubelet_pods_8df79e4a-587e-495e-9f97-54befabafc6a_volumes_kubernetes.io_secret_elasticsearch-certs', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/8df79e4a-587e-495e-9f97-54befabafc6a/volumes/kubernetes.io~secret/default-token-9cqqb' (disk '_host_mnt_paas_kubernetes_kubelet_pods_8df79e4a-587e-495e-9f97-54befabafc6a_volumes_kubernetes.io_secret_default-token-9cqqb', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/de7c0cc1-db35-48bf-a0b9-cb57d686e056/volume-subpaths/config/heartbeat/1' (disk '_host_mnt_paas_kubernetes_kubelet_pods_de7c0cc1-db35-48bf-a0b9-cb57d686e056_volume-subpaths_config_heartbeat_1', filesystem 'tmpfs', root '/..2021_02_01_13_48_32.989849910/beat.yml'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/8df79e4a-587e-495e-9f97-54befabafc6a/volume-subpaths/config/auditbeat/2' (disk '_host_mnt_paas_kubernetes_kubelet_pods_8df79e4a-587e-495e-9f97-54befabafc6a_volume-subpaths_config_auditbeat_2', filesystem 'tmpfs', root '/..2021_02_01_13_48_46.925434729/beat.yml'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/77e0b66d-5530-4bff-979d-a7fe1215b21a/volumes/kubernetes.io~secret/kafka-exporter-certs' (disk '_host_mnt_paas_kubernetes_kubelet_pods_77e0b66d-5530-4bff-979d-a7fe1215b21a_volumes_kubernetes.io_secret_kafka-exporter-certs', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/77e0b66d-5530-4bff-979d-a7fe1215b21a/volumes/kubernetes.io~secret/tom-kafka-kafka-exporter-token-646tv' (disk '_host_mnt_paas_kubernetes_kubelet_pods_77e0b66d-5530-4bff-979d-a7fe1215b21a_volumes_kubernetes.io_secret_tom-kafka-kafka-exporter-token-646tv', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/77e0b66d-5530-4bff-979d-a7fe1215b21a/volumes/kubernetes.io~secret/cluster-ca-certs' (disk '_host_mnt_paas_kubernetes_kubelet_pods_77e0b66d-5530-4bff-979d-a7fe1215b21a_volumes_kubernetes.io_secret_cluster-ca-certs', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/e40d81c0-c28d-4075-a758-79656158078d/volumes/kubernetes.io~secret/zookeeper-nodes' (disk '_host_mnt_paas_kubernetes_kubelet_pods_e40d81c0-c28d-4075-a758-79656158078d_volumes_kubernetes.io_secret_zookeeper-nodes', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/e40d81c0-c28d-4075-a758-79656158078d/volumes/kubernetes.io~secret/cluster-ca-certs' (disk '_host_mnt_paas_kubernetes_kubelet_pods_e40d81c0-c28d-4075-a758-79656158078d_volumes_kubernetes.io_secret_cluster-ca-certs', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/e40d81c0-c28d-4075-a758-79656158078d/volumes/kubernetes.io~secret/tom-kafka-zookeeper-token-87w99' (disk '_host_mnt_paas_kubernetes_kubelet_pods_e40d81c0-c28d-4075-a758-79656158078d_volumes_kubernetes.io_secret_tom-kafka-zookeeper-token-87w99', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/plugins/kubernetes.io/csi/pv/pvc-b7907da3-454f-4345-93d6-7e23dd80f326/globalmount' (disk '_host_mnt_paas_kubernetes_kubelet_plugins_kubernetes.io_csi_pv_pvc-b7907da3-454f-4345-93d6-7e23dd80f326_globalmount', filesystem 'ext4', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/e40d81c0-c28d-4075-a758-79656158078d/volumes/kubernetes.io~csi/pvc-b7907da3-454f-4345-93d6-7e23dd80f326/mount' (disk '_host_mnt_paas_kubernetes_kubelet_pods_e40d81c0-c28d-4075-a758-79656158078d_volumes_kubernetes.io_csi_pvc-b7907da3-454f-4345-93d6-7e23dd80f326_mount', filesystem 'ext4', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/be3e36ea-a563-4c2c-affe-7b997a21e543/volumes/kubernetes.io~secret/clickhouse-token-gsvvm' (disk '_host_mnt_paas_kubernetes_kubelet_pods_be3e36ea-a563-4c2c-affe-7b997a21e543_volumes_kubernetes.io_secret_clickhouse-token-gsvvm', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/plugins/kubernetes.io/csi/pv/pvc-3ed0d564-96a1-4659-bdb5-61225bf3f64b/globalmount' (disk '_host_mnt_paas_kubernetes_kubelet_plugins_kubernetes.io_csi_pv_pvc-3ed0d564-96a1-4659-bdb5-61225bf3f64b_globalmount', filesystem 'ext4', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/be3e36ea-a563-4c2c-affe-7b997a21e543/volumes/kubernetes.io~csi/pvc-3ed0d564-96a1-4659-bdb5-61225bf3f64b/mount' (disk '_host_mnt_paas_kubernetes_kubelet_pods_be3e36ea-a563-4c2c-affe-7b997a21e543_volumes_kubernetes.io_csi_pvc-3ed0d564-96a1-4659-bdb5-61225bf3f64b_mount', filesystem 'ext4', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/f6a7239c-9ffb-4974-83f3-0df266f4ed4c/volumes/kubernetes.io~secret/client-ca-cert' (disk '_host_mnt_paas_kubernetes_kubelet_pods_f6a7239c-9ffb-4974-83f3-0df266f4ed4c_volumes_kubernetes.io_secret_client-ca-cert', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/f6a7239c-9ffb-4974-83f3-0df266f4ed4c/volumes/kubernetes.io~empty-dir/ready-files' (disk '_host_mnt_paas_kubernetes_kubelet_pods_f6a7239c-9ffb-4974-83f3-0df266f4ed4c_volumes_kubernetes.io_empty-dir_ready-files', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/f6a7239c-9ffb-4974-83f3-0df266f4ed4c/volumes/kubernetes.io~secret/cluster-ca' (disk '_host_mnt_paas_kubernetes_kubelet_pods_f6a7239c-9ffb-4974-83f3-0df266f4ed4c_volumes_kubernetes.io_secret_cluster-ca', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/f6a7239c-9ffb-4974-83f3-0df266f4ed4c/volumes/kubernetes.io~secret/tom-kafka-kafka-token-vzq86' (disk '_host_mnt_paas_kubernetes_kubelet_pods_f6a7239c-9ffb-4974-83f3-0df266f4ed4c_volumes_kubernetes.io_secret_tom-kafka-kafka-token-vzq86', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/f6a7239c-9ffb-4974-83f3-0df266f4ed4c/volumes/kubernetes.io~secret/broker-certs' (disk '_host_mnt_paas_kubernetes_kubelet_pods_f6a7239c-9ffb-4974-83f3-0df266f4ed4c_volumes_kubernetes.io_secret_broker-certs', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/plugins/kubernetes.io/csi/pv/pvc-9553d9eb-b749-4234-b248-eaac097471f7/globalmount' (disk '_host_mnt_paas_kubernetes_kubelet_plugins_kubernetes.io_csi_pv_pvc-9553d9eb-b749-4234-b248-eaac097471f7_globalmount', filesystem 'ext4', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/f6a7239c-9ffb-4974-83f3-0df266f4ed4c/volumes/kubernetes.io~csi/pvc-9553d9eb-b749-4234-b248-eaac097471f7/mount' (disk '_host_mnt_paas_kubernetes_kubelet_pods_f6a7239c-9ffb-4974-83f3-0df266f4ed4c_volumes_kubernetes.io_csi_pvc-9553d9eb-b749-4234-b248-eaac097471f7_mount', filesystem 'ext4', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/9891a43b-be74-46c5-8537-bf515787a337/volumes/kubernetes.io~secret/tom-redis-bloom-token-m4k5s' (disk '_host_mnt_paas_kubernetes_kubelet_pods_9891a43b-be74-46c5-8537-bf515787a337_volumes_kubernetes.io_secret_tom-redis-bloom-token-m4k5s', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/087d494c-003e-4f5f-b194-e7cbb0157d65/volumes/kubernetes.io~secret/tom-redis-cache-token-mlp65' (disk '_host_mnt_paas_kubernetes_kubelet_pods_087d494c-003e-4f5f-b194-e7cbb0157d65_volumes_kubernetes.io_secret_tom-redis-cache-token-mlp65', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/1befeb60-7b3a-4798-baf0-ffc83e659fe9/volumes/kubernetes.io~secret/tom-redis-cache-token-mlp65' (disk '_host_mnt_paas_kubernetes_kubelet_pods_1befeb60-7b3a-4798-baf0-ffc83e659fe9_volumes_kubernetes.io_secret_tom-redis-cache-token-mlp65', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/2c46e4c3-0d32-4317-85b8-d54f2adcf8ba/volumes/kubernetes.io~secret/tom-redis-bloom-token-m4k5s' (disk '_host_mnt_paas_kubernetes_kubelet_pods_2c46e4c3-0d32-4317-85b8-d54f2adcf8ba_volumes_kubernetes.io_secret_tom-redis-bloom-token-m4k5s', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/17f41117-11fd-4442-b1c6-06b0214888b6/volumes/kubernetes.io~secret/tom-redis-cache-token-mlp65' (disk '_host_mnt_paas_kubernetes_kubelet_pods_17f41117-11fd-4442-b1c6-06b0214888b6_volumes_kubernetes.io_secret_tom-redis-cache-token-mlp65', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/d6d5d231-383c-4207-be02-99423cdbe3f1/volumes/kubernetes.io~secret/tom-redis-bloom-token-m4k5s' (disk '_host_mnt_paas_kubernetes_kubelet_pods_d6d5d231-383c-4207-be02-99423cdbe3f1_volumes_kubernetes.io_secret_tom-redis-bloom-token-m4k5s', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/ab288ec8-327c-4531-994e-300547d01396/volumes/kubernetes.io~secret/tom-redis-custom-code-token-2ffmd' (disk '_host_mnt_paas_kubernetes_kubelet_pods_ab288ec8-327c-4531-994e-300547d01396_volumes_kubernetes.io_secret_tom-redis-custom-code-token-2ffmd', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/9fae845b-1694-4515-9e93-a9b3fe18fb24/volumes/kubernetes.io~secret/tom-redis-cache-token-mlp65' (disk '_host_mnt_paas_kubernetes_kubelet_pods_9fae845b-1694-4515-9e93-a9b3fe18fb24_volumes_kubernetes.io_secret_tom-redis-cache-token-mlp65', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/6bb75194-d846-4467-a74e-c6f17d5272a9/volumes/kubernetes.io~secret/tom-redis-bloom-token-m4k5s' (disk '_host_mnt_paas_kubernetes_kubelet_pods_6bb75194-d846-4467-a74e-c6f17d5272a9_volumes_kubernetes.io_secret_tom-redis-bloom-token-m4k5s', filesystem 'tmpfs', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/plugins/kubernetes.io/csi/pv/pvc-b64b666c-16cc-45b7-bc4c-f74048f3c9f7/globalmount' (disk '_host_mnt_paas_kubernetes_kubelet_plugins_kubernetes.io_csi_pv_pvc-b64b666c-16cc-45b7-bc4c-f74048f3c9f7_globalmount', filesystem 'ext4', root '/'). (errno 13, Permission denied)
2021-02-09 05:57:09: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Cannot stat() mount point '/host/mnt/paas/kubernetes/kubelet/pods/ab288ec8-327c-4531-994e-300547d01396/volumes/kubernetes.io~csi/pvc-b64b666c-16cc-45b7-bc4c-f74048f3c9f7/mount' (disk '_host_mnt_paas_kubernetes_kubelet_pods_ab288ec8-327c-4531-994e-300547d01396_volumes_kubernetes.io_csi_pvc-b64b666c-16cc-45b7-bc4c-
...
2021-02-09 05:57:09: netdata LOG FLOOD PROTECTION too many logs (201 logs in 1 seconds, threshold is set to 200 logs in 1200 seconds). Preventing more logs from process 'netdata' for 1199 seconds.

fzyzcjy · February 9, 2021, 6:11am

I changed the permission (remove the “readonly” flag in mounts). But still have big problems: I have a lot of disks, but only see this one, and even this one has no disk space utilization.

fzyzcjy · February 9, 2021, 6:12am

There do exist one called /host/mnt, but I do not know what does it correspond to. Indeed, I have tens of PersistentVolumes, so I should at least see tens of disk_usage info.

fzyzcjy · February 9, 2021, 6:13am

The logs are as follows.

2021-02-09 06:06:44: netdata INFO  : PLUGIN[diskspace] : thread created with task id 29481
2021-02-09 06:06:44: netdata INFO  : PLUGIN[diskspace] : set name of thread 29481 to PLUGIN[diskspac
2021-02-09 06:06:45: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Mount point '/dev/termination-log' (disk '_dev_termination-log', filesystem 'ext4', root '/pods/263f16f5-5f32-47f3-9ac1-4e183ba56fb8/containers/netdata/3592feb4') is not a directory. (errno 22, Invalid argument)
2021-02-09 06:06:45: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Mount point '/etc/hosts' (disk '_etc_hosts', filesystem 'ext4', root '/pods/263f16f5-5f32-47f3-9ac1-4e183ba56fb8/etc-hosts') is not a directory.
2021-02-09 06:06:45: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Mount point '/etc/hostname' (disk '_etc_hostname', filesystem 'ext4', root '/containers/af1ed01a80a4a9ba23b50b31e387bcfd9f19398f887250a63b83cedf83ca069a/hostname') is not a directory. (errno 22, Invalid argument)
2021-02-09 06:06:45: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Mount point '/etc/resolv.conf' (disk '_etc_resolv.conf', filesystem 'ext4', root '/containers/af1ed01a80a4a9ba23b50b31e387bcfd9f19398f887250a63b83cedf83ca069a/resolv.conf') is not a directory.
2021-02-09 06:06:45: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Mount point '/run/docker.sock' (disk '_run_docker.sock', filesystem 'tmpfs', root '/docker.sock') is not a directory. (errno 22, Invalid argument)
2021-02-09 06:06:45: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Mount point '/etc/netdata/netdata.conf' (disk '_etc_netdata_netdata.conf', filesystem 'ext4', root '/pods/263f16f5-5f32-47f3-9ac1-4e183ba56fb8/volumes/kubernetes.io~configmap/config/..2021_02_09_06_06_26.713765217/netdata') is not a directory.
2021-02-09 06:06:45: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Mount point '/etc/netdata/stream.conf' (disk '_etc_netdata_stream.conf', filesystem 'ext4', root '/pods/263f16f5-5f32-47f3-9ac1-4e183ba56fb8/volumes/kubernetes.io~configmap/config/..2021_02_09_06_06_26.713765217/stream') is not a directory. (errno 22, Invalid argument)
2021-02-09 06:06:45: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Mount point '/etc/netdata/go.d.conf' (disk '_etc_netdata_go.d.conf', filesystem 'ext4', root '/pods/263f16f5-5f32-47f3-9ac1-4e183ba56fb8/volumes/kubernetes.io~configmap/config/..2021_02_09_06_06_26.713765217/go.d') is not a directory.
2021-02-09 06:06:45: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Mount point '/etc/netdata/apps_groups.conf' (disk '_etc_netdata_apps_groups.conf', filesystem 'ext4', root '/pods/263f16f5-5f32-47f3-9ac1-4e183ba56fb8/volumes/kubernetes.io~configmap/config/..2021_02_09_06_06_26.713765217/app_groups') is not a directory. (errno 22, Invalid argument)
2021-02-09 06:06:45: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Mount point '/etc/netdata/go.d/k8s_kubeproxy.conf' (disk '_etc_netdata_go.d_k8s_kubeproxy.conf', filesystem 'ext4', root '/pods/263f16f5-5f32-47f3-9ac1-4e183ba56fb8/volumes/kubernetes.io~configmap/config/..2021_02_09_06_06_26.713765217/kubeproxy') is not a directory.
2021-02-09 06:06:45: netdata ERROR : PLUGIN[diskspace] : DISKSPACE: Mount point '/etc/netdata/go.d/k8s_kubelet.conf' (disk '_etc_netdata_go.d_k8s_kubelet.conf', filesystem 'ext4', root '/pods/263f16f5-5f32-47f3-9ac1-4e183ba56fb8/volumes/kubernetes.io~configmap/config/..2021_02_09_06_06_26.713765217/kubelet') is not a directory. (errno 22, Invalid argument)

the df -h inside netdata child container:

Filesystem                Size      Used Available Use% Mounted on
/dev/mapper/docker-252:1-393217-82d2732320aba22a61688cdd6289cee8f46518215428152606dbb478fd900c6e
                          9.8G    316.9M      9.0G   3% /
tmpfs                    64.0M         0     64.0M   0% /dev
tmpfs                     7.7G         0      7.7G   0% /sys/fs/cgroup
/dev/vda1                39.1G      3.4G     33.7G   9% /host/mnt
/dev/mapper/vgpaas-kubernetes
                          9.8G    468.7M      8.8G   5% /host/mnt/paas/kubernetes/kubelet
tmpfs                     7.7G     12.0K      7.7G   0% /host/mnt/paas/kubernetes/kubelet/pods/3f849fbd-d798-40eb-9f55-76f72c41e7ed/volumes/kubernetes.io~secret/everest-csi-plugin-token-h7fkd
tmpfs                     7.7G     12.0K      7.7G   0% /host/mnt/paas/kubernetes/kubelet/pods/4ab028cc-a217-46f4-acd3-e1ea4191c0ad/volumes/kubernetes.io~secret/default-token-9cqqb
tmpfs                     7.7G     12.0K      7.7G   0% /host/mnt/paas/kubernetes/kubelet/pods/9e1c7699-9f65-480f-8f57-611d495c6906/volumes/kubernetes.io~secret/everest-csi-plugin-token-h7fkd
tmpfs                     7.7G     12.0K      7.7G   0% /host/mnt/paas/kubernetes/kubelet/pods/a9f9f061-e9ec-4aa1-8579-d0e1269d18a8/volumes/kubernetes.io~secret/default-token-gbb8r
tmpfs                     7.7G     12.0K      7.7G   0% /host/mnt/paas/kubernetes/kubelet/pods/7446443d-dcfc-4b98-a3eb-a9b133e192f8/volumes/kubernetes.io~secret/coredns-token-gc7kn
tmpfs                     7.7G     12.0K      7.7G   0% /host/mnt/paas/kubernetes/kubelet/pods/4346f106-c33a-4c70-b21c-f5f75ce2b37a/volumes/kubernetes.io~secret/strimzi-cluster-operator-token-jhgws
tmpfs                     7.7G     12.0K      7.7G   0% /host/mnt/paas/kubernetes/kubelet/pods/d888c0df-d78b-48c6-918d-4088b4bc0d0d/volumes/kubernetes.io~secret/default-token-9cqqb
tmpfs                     7.7G     12.0K      7.7G   0% /host/mnt/paas/kubernetes/kubelet/pods/6f344c2e-2316-4b94-be28-804579fcf3fa/volumes/kubernetes.io~secret/default-token-9cqqb
tmpfs                     7.7G     12.0K      7.7G   0% /host/mnt/paas/kubernetes/kubelet/pods/78a3a730-08dc-4ad7-a95f-ff02cb445222/volumes/kubernetes.io~secret/default-token-9cqqb
tmpfs                     7.7G     12.0K      7.7G   0% /host/mnt/paas/kubernetes/kubelet/pods/c015128a-2292-43ad-a6bc-00e6427b67c1/volumes/kubernetes.io~secret/default-token-9cqqb
tmpfs                     7.7G     12.0K      7.7G   0% /host/mnt/paas/kubernetes/kubelet/pods/a9959c5f-da06-4158-861e-b069fffe68bd/volumes/kubernetes.io~secret/metrics-server-token-7tq9k
tmpfs                     7.7G     12.0K      7.7G   0% /host/mnt/paas/kubernetes/kubelet/pods/1896986d-2e92-4df4-8517-4855076aee71/volumes/kubernetes.io~secret/elastic-operator-token-tzsjd
tmpfs                     7.7G      8.0K      7.7G   0% /host/mnt/paas/kubernetes/kubelet/pods/1896986d-2e92-4df4-8517-4855076aee71/volumes/kubernetes.io~secret/cert
tmpfs                     7.7G     12.0K      7.7G   0% /host/mnt/paas/kubernetes/kubelet/pods/27e60571-8aaa-4d26-b11d-c802c62105d8/volumes/kubernetes.io~secret/cceaddon-npd-token-p4rx4
tmpfs                     7.7G     12.0K      7.7G   0% /host/mnt/paas/kubernetes/kubelet/pods/28e9bc77-8922-4c08-b5bf-4529b1a6ef40/volumes/kubernetes.io~secret/default-token-9cqqb
...
/dev/sdb                  7.8G    881.1M      6.9G  11% /host/mnt/paas/kubernetes/kubelet/plugins/kubernetes.io/csi/pv/pvc-c6bd0461-dfd0-4c94-953a-5d79eaa9e523/globalmount
/dev/sdb                  7.8G    881.1M      6.9G  11% /host/mnt/paas/kubernetes/kubelet/pods/7920b528-6201-45b6-8f87-15abce67c627/volumes/kubernetes.io~csi/pvc-c6bd0461-dfd0-4c94-953a-5d79eaa9e523/mount
/dev/sda                  7.8G      1.1G      6.7G  14% /host/mnt/paas/kubernetes/kubelet/plugins/kubernetes.io/csi/pv/pvc-51a8253b-76f0-4825-9348-17650e464f42/globalmount
/dev/sda                  7.8G      1.1G      6.7G  14% /host/mnt/paas/kubernetes/kubelet/pods/89003534-ca01-4c1e-afe3-8bcfda2165bd/volumes/kubernetes.io~csi/pvc-51a8253b-76f0-4825-9348-17650e464f42/mount
/dev/sdc                  7.8G      1.2G      6.6G  15% /host/mnt/paas/kubernetes/kubelet/plugins/kubernetes.io/csi/pv/pvc-af3a108c-2f6c-4794-b09e-6bff71b93e57/globalmount
/dev/sdc                  7.8G      1.2G      6.6G  15% /host/mnt/paas/kubernetes/kubelet/pods/2f02ee9a-4479-4371-94f7-e5ea4269cc44/volumes/kubernetes.io~csi/pvc-af3a108c-2f6c-4794-b09e-6bff71b93e57/mount
...

My change in k8s yaml:

daemonset.yaml:

...
          volumeMounts:
            - name: mnt
              mountPath: /host/mnt
...
      volumes:
        - name: mnt
          hostPath:
            path: /mnt

ilyam8 · February 9, 2021, 11:14am

There do exist one called /host/mnt , but I do not know what does it correspond to.

It corresponds to /mnt of the host system (according your daemonset.yaml)

ilyam8 · February 9, 2021, 11:21am

I am not a specialist, so don’t apply this on production, test it before.

If you have some mounts inside the host /mnt and you want to see them inside the container, you could try pod.spec.containers.volumeMounts.mountPropagation, see the docs.

0 ~ $ kubectl explain pod.spec.containers.volumeMounts.mountPropagation
KIND:     Pod
VERSION:  v1

FIELD:    mountPropagation <string>

DESCRIPTION:
     mountPropagation determines how mounts are propagated from the host to
     container and the other way around. When not set, MountPropagationNone is
     used. This field is beta in 1.10.

There is HostToContainer option, perhaps it does the trick

HostToContainer - This volume mount will receive all subsequent mounts that are mounted to this volume or any of its subdirectories.

In other words, if the host mounts anything inside the volume mount, the container will see it mounted there.

vlvkobal · February 9, 2021, 11:21am

Please provide the contents of /host/proc/self/mountinfo from the inside of the netdata container.

fzyzcjy · February 9, 2021, 11:40am

@ilyam8 @vlvkobal Thanks for the replies!

If you have some mounts inside the host /mnt and you want to see them inside the container, you could try pod.spec.containers.volumeMounts.mountPropagation , see the docs.

Now it looks like:

            - name: mnt
              mountPath: /host/mnt
              mountPropagation: HostToContainer

        - name: mnt
          hostPath:
            path: /mnt

But still no luck… Nothing more appears on netdata website.

Please provide the contents of /host/proc/self/mountinfo from the inside of the netdata container.

See here: my.log · GitHub (the forum says text too long)

fzyzcjy · February 14, 2021, 6:27am

Hi @vlvkobal @ilyam8 I wonder where there are any updates? IMHO this is an important feature because it is common to monitor whether the disks are full. Thanks!

ilyam8 · April 2, 2021, 4:46pm

@fzyzcjy i completely agree that the feature is important, we will try to find the solution. It is not clear at the moment how to statfs mountpoint which are not mounted into the netdata container.

fzyzcjy · April 5, 2021, 1:25pm

@ilyam8 Looking forward to see it solved!

Topic		Replies	Views
Not seeing disk space for multiple drives Help	1	553	February 13, 2023
Problems diskspace no monitoring Help agent , collectors , configuration	3	1266	May 9, 2022
Problem with disk space metric Help cloud	2	600	January 22, 2021
Dockerized netdata does not show additional mounted disk Help agent , configuration , installation	0	407	October 8, 2023
Disk Space Monitoring Help agent	0	451	May 31, 2021