Hopefully this will help anyone else who runs into this issue
Environment:
Ubuntu 22.10
netdata v1.38.0-432-nightly
Install type: binpkg-deb
I just made a change to my ping module/collector config file to add a couple of new hosts but after I rebooted the netdata service none of the hosts showed new data (the old stuff was there).
So I backed out that change but it was still broken.
So I ran the troubleshooting commands and got an access denied:
luis@ns1:/usr/libexec/netdata/plugins.d$ sudo -u netdata -s
netdata@ns1:/usr/libexec/netdata/plugins.d$ ./go.d.plugin -d -m ping
bash: ./go.d.plugin: Permission denied
I then applied some permissions as follows:
sudo chmod a+rX /usr/libexec
sudo chmod a+rX /etc/netdata
sudo chown -R netdata:netdata /usr/libexec/netdata/
sudo chmod 0755 /usr/share/netdata/
sudo chmod -R 0775 /var/cache/netdata/
sudo systemctl restart netdata.service
And now when the agent starts I see this in the logs:
listen ip4:icmp : socket: operation not permitted
listen ip6:ipv6-icmp : socket: operation not permitted
Now, the ping docs page indicate that permissions are required for netdata to send raw ICMP ping.
The docs say to run the setcap command against the plugin directory and so I ran getcap to see if that would indicate if the setting was already there and the command gave me no output:
luis@ns1:~$ sudo getcap -vr /usr/libexec/netdata/plugins.d/go.d.plugin
/usr/libexec/netdata/plugins.d/go.d.plugin
I noticed the numerous instances of the following in my apt history.log:
This entry is the first date it occured.
Start-Date: 2023-04-26 06:36:40
Commandline: apt-get --only-upgrade install -y netdata
Upgrade: netdata:arm64 (1.38.0-424-nightly, 1.38.0-432-nightly)
Error: Sub-process /usr/bin/dpkg returned an error code (1)
End-Date: 2023-04-26 06:36:57
I then ran:
luis@ns1:~$ sudo dpkg --configure -a
[sudo] password for luis:
Setting up netdata (1.38.0-432-nightly) ...
Configuration file '/etc/netdata/netdata.conf'
==> Modified (by you or by a script) since installation.
==> Package distributor has shipped an updated version.
What would you like to do about it ? Your options are:
Y or I : install the package maintainer's version
N or O : keep your currently-installed version
D : show the differences between the versions
Z : start a shell to examine the situation
The default action is to keep your current version.
*** netdata.conf (Y/I/N/O/D/Z) [default=N] ?
which suggests to me that the installer may have been interrupted or fallen-over.
The ping module now looks to be working fine.
also now when I run getcap I get a proper output:
luis@ns1:~$ sudo getcap -vr /usr/libexec/netdata/plugins.d/go.d.plugin
/usr/libexec/netdata/plugins.d/go.d.plugin cap_net_admin,cap_net_raw=eip