Can't start Netdata with /home/netdata as cache directory

DeWaRs1206 · June 9, 2021, 7:25am

Environment

Ubuntu 18.04 / netdata v1.31.0-43-g83519b5df

Problem/Question

Hello community !

I have a master with around 25 nodes connected on it. I’m trying to cache the cache directory of netdata to store more metrics.
With the following config:

[global]
hostname = NETDATA-MASTER-PROD
enabled = yes
cloud base url = https://app.netdata.cloud
default memory mode = dbengine
page cache size = 32
dbengine multihost disk space = 312561
cache directory = /netdata
[web]
bind to = *

Everything works fine. But when I try to change cache directory setting to /home/netdata I have the following error:

2021-06-09 07:11:04: netdata INFO : MAIN : resources control: allowed file descriptors: soft = 65536, max = 65536
2021-06-09 07:11:04: netdata INFO : MAIN : Out-Of-Memory (OOM) score is already set to the wanted value 1000
2021-06-09 07:11:04: netdata INFO : MAIN : Adjusted netdata scheduling policy to idle (5), with priority 0.
2021-06-09 07:11:04: netdata INFO : MAIN : Running with process scheduling policy ‘idle’
2021-06-09 07:11:04: netdata INFO : MAIN : netdata started on pid 6373.
2021-06-09 07:11:04: netdata INFO : MAIN : Initializing spawn client.
2021-06-09 07:11:04: netdata INFO : MAIN : Executing /usr/libexec/netdata/plugins.d/system-info.sh
Spawn server is up.
2021-06-09 07:11:04: netdata INFO : MAIN : NETDATA_CONTAINER_OS_NAME=none
2021-06-09 07:11:04: netdata INFO : MAIN : NETDATA_CONTAINER_OS_ID=none
2021-06-09 07:11:04: netdata INFO : MAIN : NETDATA_CONTAINER_OS_ID_LIKE=none
2021-06-09 07:11:04: netdata INFO : MAIN : NETDATA_CONTAINER_OS_VERSION=none
2021-06-09 07:11:04: netdata INFO : MAIN : NETDATA_CONTAINER_OS_VERSION_ID=none
2021-06-09 07:11:04: netdata INFO : MAIN : NETDATA_CONTAINER_OS_DETECTION=none
2021-06-09 07:11:04: netdata INFO : MAIN : NETDATA_HOST_OS_NAME=Ubuntu
2021-06-09 07:11:04: netdata INFO : MAIN : NETDATA_HOST_OS_ID=ubuntu
2021-06-09 07:11:04: netdata INFO : MAIN : NETDATA_HOST_OS_ID_LIKE=debian
2021-06-09 07:11:04: netdata INFO : MAIN : NETDATA_HOST_OS_VERSION=18.04.5 LTS (Bionic Beaver)
2021-06-09 07:11:04: netdata INFO : MAIN : NETDATA_HOST_OS_VERSION_ID=18.04
2021-06-09 07:11:04: netdata INFO : MAIN : NETDATA_HOST_OS_DETECTION=/etc/os-release
2021-06-09 07:11:04: netdata INFO : MAIN : NETDATA_HOST_IS_K8S_NODE=false
2021-06-09 07:11:04: netdata INFO : MAIN : NETDATA_SYSTEM_KERNEL_NAME=Linux
2021-06-09 07:11:04: netdata INFO : MAIN : NETDATA_SYSTEM_KERNEL_VERSION=4.15.0-143-generic
2021-06-09 07:11:04: netdata INFO : MAIN : NETDATA_SYSTEM_ARCHITECTURE=x86_64
2021-06-09 07:11:04: netdata INFO : MAIN : NETDATA_SYSTEM_VIRTUALIZATION=none
2021-06-09 07:11:04: netdata INFO : MAIN : NETDATA_SYSTEM_VIRT_DETECTION=systemd-detect-virt
2021-06-09 07:11:04: netdata INFO : MAIN : NETDATA_SYSTEM_CONTAINER=none
2021-06-09 07:11:04: netdata INFO : MAIN : NETDATA_SYSTEM_CONTAINER_DETECTION=systemd-detect-virt
2021-06-09 07:11:04: netdata INFO : MAIN : NETDATA_SYSTEM_CPU_LOGICAL_CPU_COUNT=8
2021-06-09 07:11:04: netdata INFO : MAIN : NETDATA_SYSTEM_CPU_VENDOR=GenuineIntel
2021-06-09 07:11:04: netdata INFO : MAIN : NETDATA_SYSTEM_CPU_MODEL=Intel(R) Xeon(R) CPU E3-1270 v6 @ 3.80GHz
2021-06-09 07:11:04: netdata INFO : MAIN : NETDATA_SYSTEM_CPU_FREQ=4200000000
2021-06-09 07:11:04: netdata INFO : MAIN : NETDATA_SYSTEM_CPU_DETECTION=lscpu
2021-06-09 07:11:04: netdata INFO : MAIN : NETDATA_SYSTEM_TOTAL_RAM=33400356864
2021-06-09 07:11:04: netdata INFO : MAIN : NETDATA_SYSTEM_RAM_DETECTION=procfs
2021-06-09 07:11:04: netdata INFO : MAIN : NETDATA_SYSTEM_TOTAL_DISK_SIZE=0
2021-06-09 07:11:04: netdata INFO : MAIN : NETDATA_SYSTEM_DISK_DETECTION=sysfs
2021-06-09 07:11:04: netdata ERROR : MAIN : Failed to initialize database at /home/netdata/netdata-meta.db, due to “unable to open database file”
2021-06-09 07:11:04: netdata FATAL : netdata : Failed to initialize SQLite # : Success
2021-06-09 07:11:04: netdata INFO : MAIN : /usr/libexec/netdata/plugins.d/anonymous-statistics.sh ‘FATAL’ ‘netdata:netdata’ ‘0659@database/r:rrd_init /0’
2021-06-09 07:11:05: netdata ERROR : MAIN : Execution of anonymous statistics script returned http code 400.
2021-06-09 07:11:05: netdata INFO : MAIN : EXIT: netdata prepares to exit with code 1…
2021-06-09 07:11:05: netdata INFO : MAIN : /usr/libexec/netdata/plugins.d/anonymous-statistics.sh ‘EXIT’ ‘ERROR’ ‘-’
2021-06-09 07:11:05: netdata ERROR : MAIN : Execution of anonymous statistics script returned http code 400.
2021-06-09 07:11:05: netdata INFO : MAIN : EXIT: cleaning up the database…
2021-06-09 07:11:05: netdata INFO : MAIN : Cleaning up database [0 hosts(s)]…
2021-06-09 07:11:05: netdata INFO : MAIN : EXIT: removing netdata PID file ‘/var/run/netdata/netdata.pid’…
2021-06-09 07:11:05: netdata INFO : MAIN : EXIT: all done - netdata is now exiting - bye bye…
EOF found in spawn pipe.

The chmod for my folder seems right:
drwxr-xr-x 2 netdata netdata 4096 Jun 9 07:10 netdata/
I updated my service file to create the folder automatically:

[Service]
Type=simple
User=netdata
Group=netdata
RuntimeDirectory=netdata
RuntimeDirectoryMode=0775
PIDFile=/var/run/netdata/netdata.pid
ExecStart=/usr/sbin/netdata -P /var/run/netdata/netdata.pid -D
ExecStartPre=/bin/mkdir -p /var/cache/netdata
ExecStartPre=/bin/chown -R netdata:netdata /var/cache/netdata
ExecStartPre=/bin/mkdir -p /var/run/netdata
ExecStartPre=/bin/chown -R netdata:netdata /var/run/netdata
ExecStartPre=/bin/mkdir -p /home/netdata
ExecStartPre=/bin/chown -R netdata:netdata /home/netdata
ExecStopPost=/usr/libexec/netdata/plugins.d/reset_netdata_trace.sh
PermissionsStartOnly=true

What I expected to happen

I would like to be able to change the cache directory to /home/netdata as this is where I have some disk space

Stelios_Fragkakis · June 9, 2021, 7:58am

Hi @DeWaRs1206

Make sure that the /home directory has x permission for other

try

cd /home
chmod o+x .

and restart the agent

DeWaRs1206 · June 10, 2021, 9:14am

Hi @Stelios_Fragkakis

Thanks for your answer.
Unfortunately, it was already the case:

root@ns3117200:~# cd /home/
root@ns3117200:/home# ll
total 40
drwxr-xr-x 7 root root 4096 Jun 9 07:10 ./
drwxr-xr-x 24 root root 4096 Jun 9 07:26 …/
drwxr-xr-x 4 root root 4096 Jun 9 07:00 back/
drwx------ 2 root root 16384 May 11 09:58 lost+found/
drwxr-xr-x 2 netdata netdata 4096 Jun 9 07:10 netdata/
drwxr-xr-x 32 root root 4096 Jun 3 14:02 netdata_source/
root@ns3117200:/home# chmod o+x .
root@ns3117200:/home# ll
total 40
drwxr-xr-x 7 root root 4096 Jun 9 07:10 ./
drwxr-xr-x 24 root root 4096 Jun 9 07:26 …/
drwxr-xr-x 4 root root 4096 Jun 9 07:00 back/
drwx------ 2 root root 16384 May 11 09:58 lost+found/
drwxr-xr-x 2 netdata netdata 4096 Jun 9 07:10 netdata/
drwxr-xr-x 32 root root 4096 Jun 3 14:02 netdata_source/

Any other idea?

DeWaRs1206 · June 10, 2021, 9:16am

The disk is a mounted disk:
/dev/md4 on /home type ext4 (rw,relatime,data=ordered)

That’s the only difference I can see compare to the “/” root folder

Stelios_Fragkakis · June 10, 2021, 3:24pm

Hi @DeWaRs1206

Looks like it is the ProtectHome=read-only option in the service.file that causes this. You will need to change that, or mount the cache directory in a place other than /home

DeWaRs1206 · June 10, 2021, 3:45pm

Yes !!! Thanks a lot ! That was the issue !

Topic		Replies	Views
Cannot install netdata from source (the source directory does not include netdata-installer.sh). Leaving all files in /tmp/netdata-kickstart-L0Qm7RAAZ F Help agent-installation , agent	3	1313	January 26, 2021
Can not start netdata since upgrade to v 1.45.2 Help	2	178	April 10, 2024
Issue in netdata as container in cloud Help container-monitoring , agent	8	3802	September 14, 2020
Installation problem: creating /var/cache/netdata Help agent-installation , agent	2	1143	April 15, 2021
Unable to start netdata after installing latest nightly build Help agent	3	566	August 24, 2022

Can't start Netdata with /home/netdata as cache directory

Environment

Problem/Question

What I expected to happen

Related topics