Because the netdata version present in the package system of “Ubuntu 18.04.4 LTS” is outdated and is not updated anymore, I have installed with
“wget -O /tmp/netdata-kickstart.sh https://my-netdata.io/kickstart.sh && sh /tmp/netdata-kickstart.sh --reinstall-clean --static-only”
installed the latest version. netdata v1.40.0-181-g07bee1025
This actually worked, both local:19999 and in the netdata cloud the new version shows up and diligently produces charts. However, there are a few oddities:
There should be a netdata under /usr/sbin/ so that /etc/init.d/netdata start/status/stop still works, wasn’t it?
There is nothing of netdata in /usr/sbin.
Then I notice that “Containers & VMs” is completely missing, don’t know if this is an extra problem. Enough to display would be there anyway.
some extra information in error.log
2023-07-17 10:25:08: netdata ERROR : ACLK_MAIN : Cloud returned EC="GYUHC4NbKB-1485933", Msg-Key:"ErrAgentAlreadyConnected", Msg:"delay retry 5m0s: agent already connected", BlockRetry:false, Backoff:300s (-1 unset by cloud) (errno 2, No such file or directory)
2023-07-17 10:25:08: netdata ERROR : ACLK_MAIN : Error getting challenge
2023-07-17 10:25:08: netdata ERROR : ACLK_MAIN : Error passing Challenge/Response to get OTP
2023-07-17 10:29:00: netdata ERROR : MAIN : Cannot adjust netdata scheduling policy to batch (3), with priority 0. Falling back to nice. (errno 38, Function not implemented)
2023-07-17 10:29:00: netdata ERROR : MAIN : Cannot get my current process scheduling policy. (errno 38, Function not implemented)
2023-07-17 10:29:00: netdata ERROR : PD[perf] : child pid 24764 exited with code 1.
2023-07-17 10:29:00: netdata ERROR : PD[perf] : PLUGINSD: 'host:vb02', '/opt/netdata/usr/libexec/netdata/plugins.d/perf.plugin' (pid 24764) exited with error code 1 and haven't collected any data. Disabling it. (errno 22, Invalid argument)
2023-07-17 10:29:00: netdata ERROR : PD[ioping] : child pid 24767 exited with code 1.
2023-07-17 10:29:00: netdata ERROR : PD[ioping] : PLUGINSD: 'host:vb02', '/opt/netdata/usr/libexec/netdata/plugins.d/ioping.plugin' (pid 24767) exited with error code 1 and haven't collected any data. Disabling it. (errno 22, Invalid argument)
2023-07-17 10:29:01: netdata ERROR : ACLK_MAIN : ACLK_OTP Challenge HTTP code not 200 OK (got 409) (errno 22, Invalid argument)
2023-07-17 10:29:01: netdata ERROR : ACLK_MAIN : Cloud returned EC="FNBNohKW13-1490230", Msg-Key:"ErrAgentAlreadyConnected", Msg:"delay retry 5m0s: agent already connected", BlockRetry:false, Backoff:300s (-1 unset by cloud) (errno 2, No such file or directory)
2023-07-17 10:29:01: netdata ERROR : ACLK_MAIN : Error getting challenge
2023-07-17 10:29:01: netdata ERROR : ACLK_MAIN : Error passing Challenge/Response to get OTP
./netdatacli aclk-state
ACLK Available: Yes
ACLK Version: 2
Protocols Supported: Protobuf
Protocol Used: Protobuf
MQTT Version: 5
Claimed: Yes
Claimed Id: b29ebe62-a91d-11ed-ae12-XYZ
Cloud URL: https://app.netdata.cloud
Online: No
Reconnect count: 0
Banned By Cloud: No
Next Connection Attempt At: 2023-07-17 10:49:04
Last Backoff: 300.000
But not sure if it helps. If you do need more infos let me know.
Yes it is installed below /opt
the log was /opt/netdata/var/log/netdata/error.log
Yes the cloud is working.
I tried to copy the old dbengine contents, this results in Alerts (2) netdata.dbengine_global_errors 10min_dbengine_global_io_errors 20 errors
but this is a minor issue.
How about the >
There should be a netdata under /usr/sbin/ so that /etc/init.d/netdata start/status/stop still works, wasn’t it?
There is nothing of netdata in /usr/sbin.
shall I copy the /opt/netdata/bin/srv/netdata to /usr/sbin ?
I’m not sure, but doesn’t Ubuntu 18.04 use systemd? The kickstart script should have installed a proper systemd unit file to launch it from /opt. What are the contents of /lib/systemd/system/netdata.service ?
Then you should just need to use systemctl start netdata.
Fortunately, I have an almost identical server. This is a virtualbox, but same operating system and again netdata is under /opt.
“Ubuntu 18.04.6 LTS”
netdata v1.40.0-181-g07bee1025
~# service netdata status
● netdata.service - Real time performance monitoring
Loaded: loaded (/lib/systemd/system/netdata.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2023-07-17 08:22:00 CEST; 1 day 1h ago
Process: 344 ExecStartPre=/bin/chown -R netdata /run/netdata (code=exited, status=0/SUCCESS)
Process: 343 ExecStartPre=/bin/mkdir -p /run/netdata (code=exited, status=0/SUCCESS)
Process: 341 ExecStartPre=/bin/chown -R netdata /opt/netdata/var/cache/netdata (code=exited, status=0/SUCCESS)
Process: 331 ExecStartPre=/bin/mkdir -p /opt/netdata/var/cache/netdata (code=exited, status=0/SUCCESS)
Main PID: 345 (netdata)
Tasks: 106 (limit: 4915)
CGroup: /system.slice/netdata.service
├─ 345 /opt/netdata/bin/srv/netdata -P /run/netdata/netdata.pid -D
├─ 355 /opt/netdata/bin/srv/netdata --special-spawn-server
├─ 579 /opt/netdata/usr/libexec/netdata/plugins.d/apps.plugin 1
├─ 611 /opt/netdata/usr/libexec/netdata/plugins.d/go.d.plugin 1
├─ 635 /opt/netdata/usr/libexec/netdata/plugins.d/ebpf.plugin 1
├─17805 /opt/netdata/usr/libexec/netdata/plugins.d/debugfs.plugin 1
├─17931 /opt/netdata/usr/libexec/netdata/plugins.d/nfacct.plugin 1
└─22348 bash /opt/netdata/usr/libexec/netdata/plugins.d/tc-qos-helper.sh 1
Jul 18 08:22:00 gvcs1 [635]: thread with task id 673 finished
Jul 18 08:22:00 gvcs1 [635]: thread with task id 670 finished
Jul 18 08:22:00 gvcs1 [635]: thread with task id 668 finished
Jul 18 08:22:01 gvcs1 [635]: thread with task id 665 finished
Jul 18 08:22:01 gvcs1 [635]: thread with task id 669 finished
Jul 18 08:22:01 gvcs1 [635]: thread with task id 667 finished
Jul 18 08:22:01 gvcs1 [635]: thread with task id 666 finished
Jul 18 08:22:02 gvcs1 [635]: thread with task id 675 finished
Jul 18 08:22:06 gvcs1 [17805]: Zswap is disabled
Jul 18 08:22:15 gvcs1 [2641]: NFACCT process exiting
my problem is this one Ubuntu 18.04.4 LTS
Watch: netdatactl is not in the PATH
~# /opt/netdata/bin/netdatacli version
netdata v1.40.0-181-g07bee1025
service netdata start
Failed to start netdata.service: Unit netdata.service is masked.
# service netdata status
● netdata.service
Loaded: masked (/dev/null; bad)
Active: inactive (dead) since Mon 2023-07-17 08:45:11 CEST; 1 day 1h ago
Main PID: 38021 (code=exited, status=0/SUCCESS)
Jul 16 03:59:12 vb2 nfacct.plugin[41978]: NFACCT process exiting
Jul 16 07:59:14 vb2 nfacct.plugin[27571]: NFACCT process exiting
Jul 16 11:59:17 vb2 nfacct.plugin[11904]: NFACCT process exiting
Jul 16 15:59:19 vb2 nfacct.plugin[44930]: NFACCT process exiting
Jul 16 19:59:21 vb2 nfacct.plugin[29060]: NFACCT process exiting
Jul 16 23:59:23 vb2 nfacct.plugin[13336]: NFACCT process exiting
Jul 17 03:59:25 vb2 nfacct.plugin[46309]: NFACCT process exiting
Jul 17 07:59:27 vb2 nfacct.plugin[31879]: NFACCT process exiting
Jul 17 08:45:06 vb2 systemd[1]: Stopping Real time performance monitoring...
Jul 17 08:45:11 vb2 systemd[1]: Stopped Real time performance monitoring.
root@vbhost02:/opt/netdata# usr/libexec/netdata/netdata-updater.sh
Mi 19. Jul 08:40:46 CEST 2023 : INFO: netdata-updater.sh: Checking if a newer version of the updater script is available.
Mi 19. Jul 08:40:46 CEST 2023 : INFO: netdata-updater.sh: Downloading newest version of updater script.
Mi 19. Jul 08:40:47 CEST 2023 : INFO: netdata-updater.sh: Entering /tmp/netdata-updater-8sKxHlZPez
Mi 19. Jul 08:40:48 CEST 2023 : INFO: netdata-updater.sh: Current Version: 00104000000181
Mi 19. Jul 08:40:48 CEST 2023 : INFO: netdata-updater.sh: Latest Version: 00104000000241
Mi 19. Jul 08:40:48 CEST 2023 : INFO: netdata-updater.sh: Update available
^
|.-. .-. .-. .-. . netdata
| '-' '-' '-' '-' real-time performance monitoring, done right!
+----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+--->
(C) Copyright 2017-2023, Costa Tsaousis
All rights reserved
Released under GPL v3+
You are about to install netdata to this system.
netdata will be installed at:
/opt/netdata
The following changes will be made to your system:
# USERS / GROUPS
User 'netdata' and group 'netdata' will be added, if not present.
# LOGROTATE
This file will be installed if logrotate is present.
- /etc/logrotate.d/netdata
# SYSTEM INIT
If a supported init system is detected, appropriate configuration will be
installed to allow Netdata to run as a system service. We currently support
systemd, OpenRC, LSB init scripts, and traditional init.d setups, as well as
having experimental support for runit.
This package can also update a netdata installation that has been
created with another version of it.
Your netdata configuration will be retained.
After installation, netdata will be (re-)started.
netdata re-distributes a lot of open source software components.
Check its full license at:
https://github.com/netdata/netdata/blob/master/LICENSE
Creating directory /opt/netdata
Verifying archive integrity... 100% MD5 checksums are OK. All good.
Uncompressing netdata, the real-time performance and health monitoring system 100%
--- Attempt to create user/group netdata/netadata ---
Group 'netdata' already exists.
User 'netdata' already exists.
--- Add user netdata to required user groups ---
Group 'docker' does not exist.
Group 'nginx' does not exist.
Group 'varnish' does not exist.
Group 'haproxy' does not exist.
User 'netdata' is already in group 'adm'.
Group 'nsd' does not exist.
User 'netdata' is already in group 'proxy'.
Group 'squid' does not exist.
Group 'ceph' does not exist.
Group 'nobody' does not exist.
Group 'I2C' does not exist.
--- Install logrotate configuration for netdata ---
[/opt/netdata]# chmod 644 /etc/logrotate.d/netdata
OK ''
--- Telemetry configuration ---
You can opt out from anonymous statistics via the --disable-telemetry option, or by creating an empty file /opt/netdata/etc/netdata/.opt-out-from-anonymous-statistics
--- Install netdata at system init ---
Installing systemd service...
Synchronizing state of netdata.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable netdata
Failed to enable unit: Unit file /etc/systemd/system/netdata.service is masked.
WARNING: Failed to enable Netdata service.
--- Install (but not enable) netdata updater tool ---
cat: /system/systemd/netdata-updater.timer: No such file or directory
cat: /system/systemd/netdata-updater.service: No such file or directory
Update script is located at /opt/netdata/usr/libexec/netdata/netdata-updater.sh
--- creating quick links ---
[/opt/netdata]# rm sbin
OK ''
[/opt/netdata]# ln -s bin sbin
OK ''
[/opt/netdata/usr]# rm bin
OK ''
[/opt/netdata/usr]# ln -s ../bin bin
OK ''
[/opt/netdata/usr]# rm sbin
OK ''
[/opt/netdata/usr]# ln -s ../bin sbin
OK ''
[/opt/netdata/usr]# rm local
OK ''
[/opt/netdata/usr]# ln -s . local
OK ''
[/opt/netdata]# rm netdata-configs
OK ''
[/opt/netdata]# ln -s etc/netdata netdata-configs
OK ''
[/opt/netdata]# rm netdata-web-files
OK ''
[/opt/netdata]# ln -s usr/share/netdata/web netdata-web-files
OK ''
[/opt/netdata]# rm netdata-plugins
OK ''
[/opt/netdata]# ln -s usr/libexec/netdata netdata-plugins
OK ''
[/opt/netdata]# rm netdata-dbs
OK ''
[/opt/netdata]# ln -s var/lib/netdata netdata-dbs
OK ''
[/opt/netdata]# rm netdata-metrics
OK ''
[/opt/netdata]# ln -s var/cache/netdata netdata-metrics
OK ''
[/opt/netdata]# rm netdata-logs
OK ''
[/opt/netdata]# ln -s var/log/netdata netdata-logs
OK ''
[/opt/netdata/etc/netdata]# rm orig
OK ''
[/opt/netdata/etc/netdata]# ln -s ../../usr/lib/netdata/conf.d orig
OK ''
--- fix permissions ---
[/opt/netdata]# chmod g+rx,o+rx /opt
OK ''
[/opt/netdata]# find /opt/netdata -type d -exec chmod go+rx {} +
OK ''
[/opt/netdata]# chown -R netdata:netdata /opt/netdata/var
OK ''
[/opt/netdata]# chown -R root:netdata /opt/netdata/usr/libexec/netdata/plugins.d/ebpf.d
OK ''
--- changing plugins ownership and permissions ---
[/opt/netdata]# chown root:netdata usr/libexec/netdata/plugins.d/apps.plugin
OK ''
[/opt/netdata]# chown root:netdata usr/libexec/netdata/plugins.d/perf.plugin
OK ''
[/opt/netdata]# chown root:netdata usr/libexec/netdata/plugins.d/slabinfo.plugin
OK ''
[/opt/netdata]# chown root:netdata usr/libexec/netdata/plugins.d/debugfs.plugin
OK ''
[/opt/netdata]# chown root:netdata usr/libexec/netdata/plugins.d/ioping
OK ''
[/opt/netdata]# chown root:netdata usr/libexec/netdata/plugins.d/cgroup-network
OK ''
[/opt/netdata]# chown root:netdata usr/libexec/netdata/plugins.d/local-listeners
OK ''
[/opt/netdata]# chown root:netdata usr/libexec/netdata/plugins.d/ebpf.plugin
OK ''
[/opt/netdata]# chown root:netdata usr/libexec/netdata/plugins.d/nfacct.plugin
OK ''
[/opt/netdata]# chown root:netdata usr/libexec/netdata/plugins.d/python.d.plugin
OK ''
[/opt/netdata]# chown root:netdata usr/libexec/netdata/plugins.d/charts.d.plugin
OK ''
[/opt/netdata]# chown root:netdata usr/libexec/netdata/plugins.d/go.d.plugin
OK ''
[/opt/netdata]# chown root:netdata usr/libexec/netdata/plugins.d/ioping.plugin
OK ''
[/opt/netdata]# chown root:netdata usr/libexec/netdata/plugins.d/cgroup-network-helper.sh
OK ''
[/opt/netdata]# setcap cap_dac_read_search,cap_sys_ptrace=ep usr/libexec/netdata/plugins.d/apps.plugin
OK ''
[/opt/netdata]# setcap cap_dac_read_search=ep usr/libexec/netdata/plugins.d/slabinfo.plugin
OK ''
[/opt/netdata]# setcap cap_dac_read_search=ep usr/libexec/netdata/plugins.d/debugfs.plugin
OK ''
[/opt/netdata]# setcap cap_sys_admin=ep usr/libexec/netdata/plugins.d/perf.plugin
OK ''
[/opt/netdata]# setcap cap_net_admin,cap_net_raw=eip usr/libexec/netdata/plugins.d/go.d.plugin
OK ''
[/opt/netdata]# chmod 4750 usr/libexec/netdata/plugins.d/ioping
OK ''
[/opt/netdata]# chmod 4750 usr/libexec/netdata/plugins.d/cgroup-network
OK ''
[/opt/netdata]# chmod 4750 usr/libexec/netdata/plugins.d/local-listeners
OK ''
[/opt/netdata]# chmod 4750 usr/libexec/netdata/plugins.d/ebpf.plugin
OK ''
[/opt/netdata]# chmod 4750 usr/libexec/netdata/plugins.d/nfacct.plugin
OK ''
Configure TLS certificate paths
Using /etc/ssl for TLS configuration and certificates
Save install options
--- starting netdata ---
--- Restarting netdata instance ---
Stopping all netdata threads
[/opt/netdata]# stop_all_netdata
OK ''
Starting netdata using command 'systemctl start netdata'
[/opt/netdata]# systemctl start netdata
Failed to start netdata.service: Unit netdata.service is masked.
FAILED ''
Attempting another netdata start using command 'systemctl start netdata'
[/opt/netdata]# systemctl start netdata
Failed to start netdata.service: Unit netdata.service is masked.
FAILED ''
WARNING Netdata service still not started, attempting another forced restart by running '/opt/netdata/bin/netdata '
[/opt/netdata]# stop_all_netdata
OK ''
[/opt/netdata]# /opt/netdata/bin/netdata
OK ''
^
|.-. .-. .-. .-. .-. . netdata .-. .-. .-. .-. .-. .-
| '-' '-' '-' '-' '-' '-' '-' '-' '-' '-'
+----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+--->
[/opt/netdata]# chmod 0644 /opt/netdata/etc/netdata/netdata.conf
OK ''
Mi 19. Jul 08:41:06 CEST 2023 : INFO: netdata-updater.sh: Switching back to /tmp/netdata-updater-7VaY6jJM4G
root@vbhost02:/opt/netdata# bin/netdatacli version
netdata v1.40.0-241-g3f82ffbb7
The cloud is working continous. showing old version v1.40.0-181
local on 19999 too.
So the netdata could not restartet
(as reported obove) How about force deinstallation, but saving the claimed keys and try a fresh install?
to make the story short.
made a new try after reboot the host. Did install with the provided wget url.
After setting systemctl unmask netdata.service and killing all netdata processes I was able to use
service netdata start/status/stop
Now it is running.
I did yesterday a restore of the dbengine whitch I saved bevor the static update installation begun, with seems the reason for this: