Netdata Community

How to monitor disk space on another filesystem

Hi!

I am liking Netdata very much for monitoring my systems, so thank you for it. One particular great thing of Netdata is that it really detects a lot of things about my system and in fact I am learning from it a lot of new things to watch out for.

The flip side of this is that if something is missing, it is more difficult to understand what is the problem and how to fix it. I tried to read the documentation, but I am still quite confused.

Case in point, my system has two “main” mounted filesystems: / and /media/something. I would like to monitor the available space on both of them. Instead Netdata by default monitors the available space and inodes in /tmp, /var and /var/tmp (all of which are on the same filesystem, so have the same stats; but this is not a big problem), but nothing in /media/something. How can I add /media/something to the monitored filesystems for available space and inodes?

(on the other hand, all physical disks are detected for monitoring IO performance, so that’s ok)

Thanks for the help and the explanation!

Hello @giomasce ,

You do not need to say sorry, any problem that you have , please, let us you know. It was a pleasure to help you.

Best regards!

Hi Thiago, I can confirm that installing the last version of Netdata from your repository fixes the problem, meaning that I can now see the disk space graph. So apparently the only problem is that I was running an older version of Netdata.

Thanks again for your patience, and sorry for not noting immediately that I was using an older version.

All the best to you and to Netdata!

1 Like

With your answers, it is more clear what is happening. :slight_smile:

Netdata 1.12 was released more than one year ago, if you prefer to use deb packages to install Netdata, you can use our repository(https://packagecloud.io/netdata/netdata) and you can find more details here(https://learn.netdata.cloud/docs/agent/packaging/installer/methods/packages, ).

The memory mode that I cited could be part of the problem after Netdata 1.16, considering that you are running an older, I suppose that proc.plugin was in a different stage that time.

Your last paragraph let clear what was the problem for you. Today I fixed the problems that I was having to mount USB devices inside my VM and I could use an USB device and mount it on /media/USB. I had all charts for both local device (sda) and the USB device(sdb), so it looks like that the latest version does not have the problems that you are reporting.

I do not know your company policies, but if you are allowed to add a new repository, could you try to use the latest stable Netdata?

A last information, I did not install graphic mode on my VM, so everything that I did was with the user root and my USB device was not encrypted.

If after the update the problem persists, please, let me know.

Best regards!

@Thiago-Marques-0 said in How to monitor disk space on another filesystem:

I did everything using the newest Netdata compiled from the source. Considering everything I executed, I have few doubts for you:

1 - How did you install your Netdata?
2 - What is your Netdata version?
3 - Are you using the default configuration? Or did you change the parameter memory mode?

Hi again, thanks for going that deep with me! :slight_smile:

As for your doubts, I am using version of Netdata packaged in Debian stable, which is 1.12.0-1+deb10u1 (the Debian suffix referring to some Debian-specific change in the packaging). I never changed the parameter memory mode. I see this is older than the current Netdata release, so sorry for not having said this immediately.

BTW, my original question was maybe not completely clear, but the manually mounted disk /media/GioBackup is actually detected and visible in Netdata’s sidebar, but it only shows IO performance related metrics. What I would like to see is also disk space metrics (and corresponding alarms when available space becomes too little). I see IO performance metrics for all disks, but space metrics only for /tmp, /var, and /var/tmp.

@giomasce I tried to recreate the problem that you are having now, and I did not have success. I did the following steps:

1 - I created a VM with Debian 10 and kernel 4.19.132-1.
2 - I make a new disk (dd if=/dev/zero of=/testing)
3 - cfdisk /testing
4 - mkfs.ext4 /testing

After to have a disk that I could mount and umount whenever I needed.

5 - I started Netdata

When I access the dashboard all the disks were there. I closed the dashboard

6 - After few minutes I ran : mount /testing /mnt
7 - I access the dashboard again.

The disk was there, it was identified to /mnt

8 - I closed dashboard and I ran umount /mnt
9 - I opened the dashboard and the disk was not more there.

I did everything using the newest Netdata compiled from the source. Considering everything I executed, I have few doubts for you:

1 - How did you install your Netdata?
2 - What is your Netdata version?
3 - Are you using the default configuration? Or did you change the parameter memory mode?

Best regards!

@giomasce firstly sorry for the delay, but I was very busy yesterday.

I will try to replicate the problem that you are having on Debian to understand the motive we have different behaviors on Slackware and Debian. It could be something related to kernels, but I cannot discard something on our packages.

Best regards!

@Thiago-Marques-0 Hi, thanks for helping!

I am using Debian stable. More precisely:

root@panopticon:~# uname -a; grep -Hv “^#” /etc/*release
Linux panopticon 4.19.0-9-amd64 #1 SMP Debian 4.19.118-2+deb10u1 (2020-06-07) x86_64 GNU/Linux
/etc/os-release:PRETTY_NAME="Debian GNU/Linux 10 (buster)"
/etc/os-release:NAME="Debian GNU/Linux"
/etc/os-release:VERSION_ID="10"
/etc/os-release:VERSION="10 (buster)"
/etc/os-release:VERSION_CODENAME=buster
/etc/os-release:ID=debian
/etc/os-release:HOME_URL="https://www.debian.org/"
/etc/os-release:SUPPORT_URL="https://www.debian.org/support"
/etc/os-release:BUG_REPORT_URL="https://bugs.debian.org/"

However, note that the filesystem under /media/something (which is actually /media/GioBackup as you can see in my previous post) is not automatically mounted. It is mounted after startup, because it is an encrypted disk, but the mounting process is done manually be me over an SSH connection.

Also, user netdata seems to be able to access /proc/diskstats (although filesystem usage information does not appear there):

root@panopticon:~# sudo -u netdata cat /proc/diskstats 
   8       0 sda 1006501 326197 103241874 6857762 233421 433763 6580216 1553013 0 5311432 5311460 0 0 0 0
   8       1 sda1 137 0 7936 2506 0 0 0 0 0 2344 2344 0 0 0 0
   8       2 sda2 1006308 326197 103230802 6854558 233231 433763 6580216 1548300 0 5326536 5326540 0 0 0 0
  11       0 sr0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   8      16 sdb 25486653 40942716 587813069 221164091 4201773 26191575 354981112 27812978 0 147216988 147217108 0 0 0 0
   8      17 sdb1 25486616 40942716 587811861 221163967 4201773 26191575 354981112 27812978 1 150196940 150196940 0 0 0 0
   7       0 loop0 41 0 244 3353 0 0 0 0 0 3264 3352 0 0 0 0
   7       1 loop1 39 0 234 3759 0 0 0 0 0 3756 3756 0 0 0 0
   7       2 loop2 58 0 2128 10503 0 0 0 0 0 5096 10488 0 0 0 0
   7       3 loop3 49333 0 100674 495474 0 0 0 0 0 22856 494552 0 0 0 0
   7       4 loop4 2 0 10 0 0 0 0 0 0 0 0 0 0 0 0
   7       5 loop5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   7       6 loop6 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   7       7 loop7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 254       0 dm-0 66429308 0 587810496 2379219676 30393348 0 354981112 1757241508 13 152444640 4230200324 0 0 0 0

Thanks again for helping!

@OdysLam I think Netdata should be able to see everything:

root@panopticon# ps aux | grep netdata
netdata  19146  1.0  1.6 179808 33408 ?        Ssl  10:36   1:34 /usr/sbin/netdata -D
netdata  19166  0.8  0.7  65184 15916 ?        Sl   10:36   1:17 /usr/bin/python3 /usr/lib/netdata/plugins.d/python.d.plugin 1
netdata  19167  1.3  0.1   4244  2860 ?        S    10:36   2:07 /usr/lib/netdata/plugins.d/apps.plugin 1
netdata  20912  0.1  0.1   3784  2872 ?        S    12:36   0:02 bash /usr/lib/netdata/plugins.d/tc-qos-helper.sh 1
root     21440  0.0  0.0   6228   884 pts/2    S+   13:13   0:00 grep netdata

So clearly Netdata is running as user netdata. Then we can see that user netdata is able to navigate the relevant filesystem and query filesystem usage:

root@panopticon:~# sudo -u netdata bash
netdata@panopticon:/root$ cd /
netdata@panopticon:/$ ls
bin  boot  borg  dev  etc  home  initrd.img  initrd.img.old  lib  lib64  lost+found  media  mnt  opt  proc  root  run  sbin  snap  srv	swap.img  sys  tmp  usr  var  vmlinuz  vmlinuz.old
netdata@panopticon:/$ cd /media/GioBackup/
netdata@panopticon:/media/GioBackup$ ls
backup_congelati  borg	cellulare  duplicity  du.txt  giovanni	lost+found  nextcloud  rsnapshot  vecchio_irene
netdata@panopticon:/media/GioBackup$ df 
File system           1K-blocchi      Usati Disponib. Uso% Montato su
udev                      990108          0    990108   0% /dev
tmpfs                     200996      11056    189940   6% /run
/dev/sda2              113815716  103270108   4694372  96% /
tmpfs                    1004976          0   1004976   0% /dev/shm
tmpfs                       5120          0      5120   0% /run/lock
tmpfs                    1004976          0   1004976   0% /sys/fs/cgroup
/dev/loop0                  4096       4096         0 100% /snap/hw-probe/828
/dev/loop2                 98944      98944         0 100% /snap/core/9804
/dev/loop3                 99456      99456         0 100% /snap/core/9993
/dev/loop1                  4096       4096         0 100% /snap/hw-probe/755
/dev/mapper/GioBackup 3844638516 3539119952 110151408  97% /media/GioBackup
tmpfs                     200992          0    200992   0% /run/user/1000
tmpfs                     200992          0    200992   0% /run/user/0

Hello @giomasce ,

The plugin that monitors the disk information is proc.plugin that is an internal plugin.

I started my computer and I did not have any USB device connected, after a while I connected the device and I observed that it was recognized by Netdata. I have to say that between the first access to Netdata and the connection of the USB, I closed Netdata dashboard and I reopen again.
Another thing that it is necessary to say, it is the fact that on my distribution(Slackware), the system is using /run/media, instead /media.

For I understand better what is happening on your computer, please, can you run the following command on it and give us the output?

uname -a; grep -Hv “^#” /etc/*release

The plugin expects to read data from /proc/diskstats, and I have to understand what your distribution is doing.

Best regards!

Hey,

Welcome to the community @giomasce, it really means a lot to us that you are enjoying the product. We can’t stress enough how important this is for us.

According to our documentation, this particular plugin requires that it has access to all the drives that it should be monitoring. By default is all, so that shouldn’t be a problem. Please ensure that netdata has the appropriate rights.

We will get to the bottom of this! :muscle: