Ah, thanks, found the containers there under cgroups, see Docker containers running on EC2 not visible in cloud.
In terms of systemd, I just spun up a fresh AmazonLinux 2 EC2 instance and ran the kickstart installer on it.
The systemd file that got installed is:
[ec2-user@ip-172-31-xx-xxx ~]$ cat /lib/systemd/system/netdata.service
# SPDX-License-Identifier: GPL-3.0-or-later
[Unit]
Description=Real time performance monitoring
# append here other services you want netdata to wait for them to start
After=network.target httpd.service squid.service nfs-server.service mysqld.service mysql.service named.service postfix.service chronyd.service
[Service]
Type=simple
User=netdata
Group=netdata
RuntimeDirectory=netdata
CacheDirectory=netdata
StateDirectory=netdata
LogsDirectory=netdata
RuntimeDirectoryMode=0775
StateDirectoryMode=0755
CacheDirectoryMode=0755
LogsDirectoryMode=2750
EnvironmentFile=-/etc/default/netdata
ExecStart=/usr/sbin/netdata -D $EXTRA_OPTS
# saving a big db on slow disks may need some time
TimeoutStopSec=150
# restart netdata if it crashes
Restart=on-failure
RestartSec=30
# Valid policies: other (the system default) | batch | idle | fifo | rr
# To give netdata the max priority, set CPUSchedulingPolicy=rr and CPUSchedulingPriority=99
CPUSchedulingPolicy=batch
# This sets the scheduling priority (for policies: rr and fifo).
# Priority gets values 1 (lowest) to 99 (highest).
#CPUSchedulingPriority=1
# For scheduling policy 'other' and 'batch', this sets the lowest niceness of netdata (-20 highest to 19 lowest).
Nice=0
[Install]
WantedBy=multi-user.target
[ec2-user@ip-172-31-xx-xxx ~]$
As you can see the paths in the file are incorrect. The EnvironmentFile and ExecStart files don’t exist:
[ec2-user@ip-172-31-xx-xxx ~]$ ls -l /etc/default/netdata
ls: cannot access /etc/default/netdata: No such file or directory
[ec2-user@ip-172-31-xx-xxx ~]$
[ec2-user@ip-172-31-xx-xxx ~]$ ls -l /usr/sbin/netdata
ls: cannot access /usr/sbin/netdata: No such file or directory
[ec2-user@ip-172-31-xx-xxx ~]$
since all the netdata files were installed under /opt/netdata.
The correct paths to point to for should have /opt/netdata pre-prended:
[ec2-user@ip-172-31-xx-xxx ~]$ ls -l /opt/netdata/usr/sbin/netdata
-rwxr-xr-x 1 netdata netdata 146 Nov 21 00:22 /opt/netdata/usr/sbin/netdata
[ec2-user@ip-172-31-xx-xxx ~]$
/etc/default/netdata does not exist but there is an /opt/netdata/etc/netdata containing configuration files:
[ec2-user@ip-172-31-xx-xxx ~]$ ls -l /opt/netdata/etc/netdata
total 32
drwxr-xr-x 2 netdata netdata 6 Nov 21 00:22 charts.d
drwxr-xr-x 2 netdata netdata 6 Nov 21 00:22 custom-plugins.d
drwxr-xr-x 2 netdata netdata 6 Nov 21 00:22 ebpf.d
-rwxr-xr-x 1 netdata netdata 2069 Nov 21 00:22 edit-config
drwxr-xr-x 2 netdata netdata 6 Nov 21 00:22 go.d
drwxr-xr-x 2 netdata netdata 6 Nov 21 00:22 health.d
-rw-r--r-- 1 root root 25824 Nov 21 15:10 netdata.conf
lrwxrwxrwx 1 netdata netdata 28 Nov 21 15:09 orig -> ../../usr/lib/netdata/conf.d
drwxr-xr-x 2 netdata netdata 6 Nov 21 00:22 python.d
drwxr-xr-x 2 netdata netdata 6 Nov 21 00:22 ssl
drwxr-xr-x 2 netdata netdata 6 Nov 21 00:22 statsd.d
[ec2-user@ip-172-31-xx-xxx ~]$
So the bug seems to be that the installer does not craft a systemd file to match where it un-tar’ed the binary and thus systemctl commands do not work and netdata won’t come up after reboot.
Interestingly, the kickstart installer does try to start it up from the correct location of the binary, but then says it can’t find the conf file (which does exist at that location) which is immediately followed by a chmod to set perms on the netdata.conf file:
[/opt/netdata]# /opt/netdata/bin/netdata
2022-11-21 15:10:37: netdata INFO : MAIN : CONFIG: cannot load user config '/opt/netdata/etc/netdata/netdata.conf'. Will try the stock version.
2022-11-21 15:10:37: netdata INFO : MAIN : CONFIG: cannot load stock config '/opt/netdata/usr/lib/netdata/conf.d/netdata.conf'. Running with internal defaults.
2022-11-21 15:10:37: netdata INFO : MAIN : CONFIG: cannot load cloud config '/opt/netdata/var/lib/netdata/cloud.d/cloud.conf'. Running with internal defaults.
2022-11-21 15:10:37: netdata INFO : MAIN : Found 0 legacy dbengines, setting multidb diskspace to 256MB
2022-11-21 15:10:37: netdata INFO : MAIN : Created file '/opt/netdata/var/lib/netdata/dbengine_multihost_size' to store the computed value
OK ''
Downloading default configuration from netdata...
[/opt/netdata]# curl -sSL --connect-timeout 10 --retry 3 http://localhost:19999/netdata.conf
OK ''
[/opt/netdata]# mv /opt/netdata/etc/netdata/netdata.conf.new /opt/netdata/etc/netdata/netdata.conf
OK ''
OK New configuration saved for you to edit at /opt/netdata/etc/netdata/netdata.conf
^
|.-. .-. .-. .-. .-. . netdata .-. .-. .-. .-. .-. .-
| '-' '-' '-' '-' '-' '-' '-' '-' '-' '-'
+----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+--->
[/opt/netdata]# chmod 0644 /opt/netdata/etc/netdata/netdata.conf
OK
which makes it look like the chmod is out of order and should be done earlier so that the perms allow the startup command to read the conf file.
Don’t know if all this is AmazonLinux 2 specific behavior or it it’s applicable to other non-native package installations as well.
-RP