Tried uninstalling netdata, it failed

Tudvari · October 25, 2023, 9:36am

Hi guys,

I always have issues with updating NetData on one of my instances and while I manage to update this time, but the version didn’t update in the cloud dashboard. Then I did a system reboot and now it’s offline.

When I did a service status it says
● netdata.service
Loaded: masked (/dev/null; bad)
Active: inactive (dead)

But it also said this both before the update and the reboot. While the at the cloud dashboard everything was looking fine both times. The instance was monitored well.

So I thought I should just do a complete reinstall, with first cleaning everything NetData related. But it failed.

root@build:~# wget -O /tmp/netdata-kickstart.sh https://my-netdata.io/kickstart.sh && sh /tmp/netdata-kickstart.sh --uninstall
–2023-10-25 11:26:31-- https://my-netdata.io/kickstart.sh
Resolving my-netdata.io (my-netdata.io)… 104.21.13.159, 172.67.156.192, 2606:4700:3031::6815:d9f, …
Connecting to my-netdata.io (my-netdata.io)|104.21.13.159|:443… connected.
HTTP request sent, awaiting response… 200 OK
Length: unspecified [application/octet-stream]
Saving to: ‘/tmp/netdata-kickstart.sh’

/tmp/netdata-kickstart.sh [ <=> ] 82.99K --.-KB/s in 0.002s

2023-10-25 11:26:31 (42.9 MB/s) - ‘/tmp/netdata-kickstart.sh’ saved [84980]

— Using /tmp/netdata-kickstart-EJL3CjGIVp as a temporary directory. —
— Checking for existing installations of Netdata… —
[/tmp/netdata-kickstart-EJL3CjGIVp]# test -x /usr/local/usr/libexec/netdata/netdata-uninstaller.sh
FAILED

— Downloading netdata-uninstaller … —
[/tmp/netdata-kickstart-EJL3CjGIVp]# /usr/bin/curl --fail -q -sSL --connect-timeout 10 --retry 3 --output /tmp/netdata-kickstart-EJL3CjGIVp/netdata-uninstaller.sh https://raw.githubusercontent.com/netdata/netdata/master/packaging/installer/netdata-uninstaller.sh
OK

[/tmp/netdata-kickstart-EJL3CjGIVp]# /tmp/netdata-kickstart-EJL3CjGIVp/netdata-uninstaller.sh --yes
/tmp/netdata-kickstart-EJL3CjGIVp/netdata-uninstaller.sh: 711: .: Can’t open /etc/netdata/.environment

Wed Oct 25 11:26:31 CEST 2023 : FATAL: netdata-uninstaller.sh: FAILED TO UNINSTALL NETDATA: Failed to completely remove Netdata from this system.
FAILED

Finished uninstalling the Netdata Agent.The following non-fatal warnings or errors were encountered:

Command “test -x /usr/local/usr/libexec/netdata/netdata-uninstaller.sh” failed with exit code 1.

Command “/tmp/netdata-kickstart-EJL3CjGIVp/netdata-uninstaller.sh --yes” failed with exit code 1.

[/root]# rm -rf /tmp/netdata-kickstart-EJL3CjGIVp
OK

The installation type is:

root@build:~# netdata -W buildinfo | grep -E ‘Installation Type|Install type:’
Installation Type __________________________________________ : kickstart-static

Here are (possibly) all netdata related folders on the drive:

root@build:~# find / -d -name netdata
/usr/lib/netdata
/usr/local/bin/netdata
/etc/logrotate.d/netdata
/etc/init.d/netdata
/etc/default/netdata
/etc/netdata
/opt/netdata/usr/libexec/netdata
/opt/netdata/usr/lib/netdata/system/freebsd/rc.d/netdata
/opt/netdata/usr/lib/netdata/system/initd/init.d/netdata
/opt/netdata/usr/lib/netdata/system/lsb/init.d/netdata
/opt/netdata/usr/lib/netdata/system/openrc/init.d/netdata
/opt/netdata/usr/lib/netdata/system/openrc/conf.d/netdata
/opt/netdata/usr/lib/netdata/system/logrotate/netdata
/opt/netdata/usr/lib/netdata
/opt/netdata/usr/share/netdata
/opt/netdata/bin/srv/netdata
/opt/netdata/bin/netdata
/opt/netdata/etc/netdata
/opt/netdata/var/lib/netdata
/opt/netdata/var/cache/netdata
/opt/netdata/var/log/netdata
/opt/netdata
/var/lib/netdata
/var/cache/netdata
/var/log/netdata

Could you help me out what’s happening? Maybe there are two netdatas conflicting each other? Thanks in advance!

Tudvari · November 15, 2023, 8:48am

bump Could someone suggest me something? Thanks in advance!

Tasos_Katsoulas · November 15, 2023, 3:09pm

Hello @Tudvari and sorry for missing your post.

Could you please share the out of ps aux | grep netdata

plus if you don’t care about the data of this Agent I would recommend to;

EDITED:
sh /opt/netdata/usr/libexec/netdata/netdata-uninstaller.sh --env /opt/netdata/etc/netdata/.environment --yes

and then try to run the kickstart with `–uninstall`` option

The rationale is; if two netdata deployments are there first uninstall the /opt/netdat and then the systemd wide

But we need to find the root cause of this, you provided a details description, but I need you to remember what actions you did in the node from the very beginning. when you noticed the problem and what actions you did then. . .

Tudvari · November 15, 2023, 3:14pm

No problem, thanks in advance for all the help!

The ps aux grep returned only the grep itself.

Sadly I don’t remember the details anymore. :\

EDIT: running the uninstall right now

Tasos_Katsoulas · November 15, 2023, 3:16pm

edited, be aware of what it prompts you to clean up, not sure if this environment file is tampered somehow. . .

We want to delete only any netdata under /opt/netdata and or the system wide elements (services, logrotate etc)

Tudvari · November 15, 2023, 3:24pm

I ran your line, I said “yes” to all prompt because they all seemed reasonable.

After that I tried running the kickstart with ‘–uninstall’ and it said the following… is this the expected?

root@build:~# wget -O /tmp/netdata-kickstart.sh https://my-netdata.io/kickstart.sh && sh /tmp/netdata-kickstart.sh --uninstall
–2023-11-15 16:21:33-- https://my-netdata.io/kickstart.sh
Resolving my-netdata.io (my-netdata.io)… <…>, …
Connecting to my-netdata.io (my-netdata.io)|<…>|:443… connected.
HTTP request sent, awaiting response… 200 OK
Length: unspecified [application/octet-stream]
Saving to: ‘/tmp/netdata-kickstart.sh’

/tmp/netdata-kickstart.sh [ <=> ] 82.82K --.-KB/s in 0.006s

2023-11-15 16:21:33 (14.0 MB/s) - ‘/tmp/netdata-kickstart.sh’ saved [84811]

— Using /tmp/netdata-kickstart-4eqMRzqASc as a temporary directory. —
— Checking for existing installations of Netdata… —
[/tmp/netdata-kickstart-4eqMRzqASc]# test -x /usr/libexec/netdata/netdata-uninstaller.sh
FAILED

— Downloading netdata-uninstaller … —
[/tmp/netdata-kickstart-4eqMRzqASc]# /usr/bin/curl --fail -q -sSL --connect-timeout 10 --retry 3 --output /tmp/netdata-kickstart-4eqMRzqASc/netdata-uninstaller.sh https://raw.githubusercontent.com/netdata/netdata/master/packaging/installer/netdata-uninstaller.sh
OK

[/tmp/netdata-kickstart-4eqMRzqASc]# /tmp/netdata-kickstart-4eqMRzqASc/netdata-uninstaller.sh --yes
/tmp/netdata-kickstart-4eqMRzqASc/netdata-uninstaller.sh: 711: .: Can’t open /etc/netdata/.environment

Wed Nov 15 16:21:34 CET 2023 : FATAL: netdata-uninstaller.sh: FAILED TO UNINSTALL NETDATA: Failed to completely remove Netdata from this system.
FAILED

Finished uninstalling the Netdata Agent.The following non-fatal warnings or errors were encountered:

Command “test -x /usr/libexec/netdata/netdata-uninstaller.sh” failed with exit code 1.

Command “/tmp/netdata-kickstart-4eqMRzqASc/netdata-uninstaller.sh --yes” failed with exit code 1.

[/root]# rm -rf /tmp/netdata-kickstart-4eqMRzqASc
OK

Tasos_Katsoulas · November 15, 2023, 3:26pm

ls -la /etc/netdata & ls -la /etc/default/netdata?

Q: this a BSD variant, did you installed netdata via ports in the first place, could you recall that???

Tudvari · November 15, 2023, 3:32pm

Sorry, I remember that there were some issues in the past regarding one of our instances’ netdata installation but nothing in particular.

root@build:~# ls -la /etc/netdata & ls -la /etc/default/netdata
[1] 4468
total 20
drwxr-xr-x 2 root root 4096 Dec 15 2022 .
drwxr-xr-x 90 root root 4096 Nov 15 16:28 …
-rwxr-xr-x 1 root root 2045 Jun 10 2022 edit-config
-rw-r–r-- 1 root root 69 Jun 10 2022 .install-type
-rw-r–r-- 1 root root 533 Jun 10 2022 netdata.conf
-rw-r–r-- 1 root root 84 Jun 10 2022 /etc/default/netdata
[1]+ Done ls -la /etc/netdata

root@build:~# cat /etc/netdata/.install-type
INSTALL_TYPE=‘binpkg-deb’
PREBUILT_ARCH=‘x86_64’
PREBUILT_DISTRO=’ ’

Tasos_Katsoulas · November 15, 2023, 3:36pm

Ok what’s your platform? try something like uname -srm

Tudvari · November 15, 2023, 3:38pm

root@build:~# uname -srm
Linux 4.9.0-19-amd64 x86_64

Tasos_Katsoulas · November 15, 2023, 3:41pm

also try an apt purge netdata, why do I believe it was a BSD variant?

Tudvari · November 15, 2023, 3:46pm

root@build:~# apt purge netdata
Reading package lists… Done
Building dependency tree
Reading state information… Done
The following packages were automatically installed and are no longer required:
libnetfilter-acct1 libprotobuf10 libsnappy1v5
Use ‘apt autoremove’ to remove them.
The following packages will be REMOVED:
netdata*
0 upgraded, 0 newly installed, 1 to remove and 1 not upgraded.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] y
(Reading database … 120795 files and directories currently installed.)
Purging configuration files for netdata (1.35.1) …
dpkg: warning: while removing netdata, directory ‘/var/log/netdata’ not empty so not removed
dpkg: warning: while removing netdata, directory ‘/var/lib/netdata’ not empty so not removed
dpkg: warning: while removing netdata, directory ‘/var/cache/netdata’ not empty so not removed
dpkg: warning: while removing netdata, directory ‘/usr/lib/netdata/conf.d/health.d’ not empty so not removed
Processing triggers for systemd (232-25+deb9u14) …

Tasos_Katsoulas · November 15, 2023, 3:48pm

force delete the directories, that the dpkg couldn’t remove.

and I think you are ready to re-deploy, keep in mind stick with --static-only installation for this node, it’s debian 9 which we don’t support native packages atm

Tudvari · November 15, 2023, 4:03pm

Removed them, and then ran the kickstart script with --static-only flag .

However when I tried to claim it, it said the following:

root@build:~# wget -O /tmp/netdata-kickstart.sh https://my-netdata.io/kickstart.sh && sh /tmp/netdata-kickstart.sh --nightly-channel --claim-token <…> --claim-rooms <…> --claim-url https://app.netdata.cloud
–2023-11-15 17:01:04-- https://my-netdata.io/kickstart.sh
Resolving my-netdata.io (my-netdata.io)… <…>, …
Connecting to my-netdata.io (my-netdata.io)<…>|:443… connected.
HTTP request sent, awaiting response… 200 OK
Length: unspecified [application/octet-stream]
Saving to: ‘/tmp/netdata-kickstart.sh’

/tmp/netdata-kickstart.sh [ <=> ] 82.82K --.-KB/s in 0.007s

2023-11-15 17:01:04 (11.9 MB/s) - ‘/tmp/netdata-kickstart.sh’ saved [84811]

— Using /tmp/netdata-kickstart-sPpTdLmMiq as a temporary directory. —
— Checking for existing installations of Netdata… —
WARNING Found an existing netdata install at /usr/local, but could not determine the install type. Usually this means you installed Netdata through your distribution’s regular package repositories or some other unsupported method.

Attempting to update an existing install with an unknown installation type is not officially supported. It may work, but it also might break your system. If you just want to claim this install, you should re-run this command with the --claim-only option instead. Are you sure you want to continue? [y/n]

Tasos_Katsoulas · November 15, 2023, 4:17pm

Probably we left behind something, please

find any netdata dir in the system
what’s the output of: ps aux | grep netdata
what’s the output of: systemctl status netdata
what’s the output of ./opt/netdata/usr/sbin/netdata -W buildinfo
what\s the output of which netdata (if you don’t have exported the /opt/netdata/usr/sbin in your paths, it should report nothing)

Tudvari · November 16, 2023, 10:07am

Hi,

So I ran that netdata-uninstaller.sh in the exact way you mentioned earlier, said yes to everything and then I did a reboot.

Here are the checks:

root@build:~# ps aux | grep netdata
root 4333 0.0 0.0 12780 964 pts/0 S+ 11:02 0:00 grep netdata
root@build:~# systemctl status netdata
Unit netdata.service could not be found.
root@build:~# ./opt/netdata/usr/sbin/netdata -W buildinfo
-bash: ./opt/netdata/usr/sbin/netdata: No such file or directory
root@build:~# which netdata
root@build:~# sudo find / -type d -name ‘netdata’ 2>/dev/null
root@build:~# sudo find / -type f -name ‘netdata’ 2>/dev/null
/netdata-updater.log.lhcyQN
/netdata-updater.log.GmG2EP
/netdata-updater.log.WUIIT1
/netdata-updater.log.jDaqH0
/netdata-updater.log.eAJnr7
/netdata-updater.log.HPkURP
/etc/apt/trusted.gpg.d/netdata-repoconfig-archive-keyring.gpg
/etc/apt/trusted.gpg.d/netdata-archive-keyring.gpg
/etc/apt/trusted.gpg.d/netdata-edge-archive-keyring.gpg
/etc/apt/sources.list.d/netdata-edge.list
/etc/apt/sources.list.d/netdata.list
/netdata-updater.log.eFYtij
/netdata-updater.log.LFLcZe
/netdata-updater.log.mMOsSw
/netdata-updater.log.WKzOWc
/var/lib/apt/lists/packagecloud.io_netdata_netdata-edge_debian_dists_stretch_InRelease
/var/lib/apt/lists/packagecloud.io_netdata_netdata-repoconfig_debian_dists_stretch_main_binary-amd64_Packages
/var/lib/apt/lists/packagecloud.io_netdata_netdata_debian_dists_stretch_InRelease
/var/lib/apt/lists/packagecloud.io_netdata_netdata_debian_dists_stretch_Release.gpg
/var/lib/apt/lists/packagecloud.io_netdata_netdata_debian_dists_stretch_Release
/var/lib/apt/lists/packagecloud.io_netdata_netdata-edge_debian_dists_stretch_main_binary-amd64_Packages
/var/lib/apt/lists/packagecloud.io_netdata_netdata-repoconfig_debian_dists_stretch_InRelease
/var/lib/apt/lists/packagecloud.io_netdata_netdata_debian_dists_stretch_main_binary-amd64_Packages
/var/lib/dpkg/info/netdata-repo-edge.list
/var/lib/dpkg/info/netdata-repo.list
/var/cache/apt/archives/netdata_1.35.1_amd64.deb
/var/cache/apt/archives/netdata_1.34.1_amd64.deb
/var/cache/apt/archives/netdata-repo_1-1_all.deb
/var/cache/apt/archives/netdata-repo-edge_1-2_all.deb
/var/cache/apt/archives/netdata_1.33.1-236-nightly_amd64.deb
/var/cache/apt/archives/netdata-repo-edge_1-1_all.deb

Am I ready to try installing a static-only build again?

Tasos_Katsoulas · November 16, 2023, 1:28pm

Ok proceed with the following steps:

Delete the outdated sources of netdata from the apt package manager (just for completeness)
Check the output of thedpkg-query --show --showformat '${Status}' "netdata", Probably it advertises that you have a netdata package, you need to get rid of it
Most of the other stuff can be gone with an apt-get clean

Tudvari · November 16, 2023, 1:43pm

Done, and it says
dpkg-query: no packages found matching netdata

File search says:

root@build:/# sudo find / -type f -name '*netdata*' 2>/dev/null
/netdata-updater.log.lhcyQN
/netdata-updater.log.GmG2EP
/netdata-updater.log.WUIIT1
/netdata-updater.log.jDaqH0
/netdata-updater.log.eAJnr7
/netdata-updater.log.HPkURP
/netdata-updater.log.eFYtij
/netdata-updater.log.LFLcZe
/netdata-updater.log.mMOsSw
/netdata-updater.log.WKzOWc
/var/lib/apt/lists/packagecloud.io_netdata_netdata-edge_debian_dists_stretch_InRelease
/var/lib/apt/lists/packagecloud.io_netdata_netdata-repoconfig_debian_dists_stretch_main_binary-amd64_Packages
/var/lib/apt/lists/packagecloud.io_netdata_netdata_debian_dists_stretch_InRelease
/var/lib/apt/lists/packagecloud.io_netdata_netdata_debian_dists_stretch_Release.gpg
/var/lib/apt/lists/packagecloud.io_netdata_netdata_debian_dists_stretch_Release
/var/lib/apt/lists/packagecloud.io_netdata_netdata-edge_debian_dists_stretch_main_binary-amd64_Packages
/var/lib/apt/lists/packagecloud.io_netdata_netdata-repoconfig_debian_dists_stretch_InRelease
/var/lib/apt/lists/packagecloud.io_netdata_netdata_debian_dists_stretch_main_binary-amd64_Packages
/var/lib/dpkg/info/netdata-repo-edge.list
/var/lib/dpkg/info/netdata-repo.list

Should I reinstall it now?

Tasos_Katsoulas · November 16, 2023, 2:08pm

dpkg-query --show --showformat '${Status}' "netdata" ,

Kickstart performs the same check and package installation under /usr/local is returned

I know that you saw a lot of find results but, could you verify that nothing related to netdata /usr/local?

Check system-wide with

sudo find / -type f -iname '*netdata*'

EDITED: Please share the output before run the kickstart again

If no netdata residues are there, do the following run the kickstart with --static-only and --reinstall-even-if-unsafe

Tudvari · November 16, 2023, 2:18pm

Sure, here are the results:

root@build:/# dpkg-query --show --showformat '${Status}' "netdata"
dpkg-query: no packages found matching netdata

root@build:/# sudo find / -type f -iname '*netdata*'
/netdata-updater.log.lhcyQN
/netdata-updater.log.GmG2EP
/netdata-updater.log.WUIIT1
/netdata-updater.log.jDaqH0
/netdata-updater.log.eAJnr7
/netdata-updater.log.HPkURP
/netdata-updater.log.eFYtij
/netdata-updater.log.LFLcZe
/netdata-updater.log.mMOsSw
/netdata-updater.log.WKzOWc
/var/lib/apt/lists/packagecloud.io_netdata_netdata-edge_debian_dists_stretch_InRelease
/var/lib/apt/lists/packagecloud.io_netdata_netdata-repoconfig_debian_dists_stretch_main_binary-amd64_Packages
/var/lib/apt/lists/packagecloud.io_netdata_netdata_debian_dists_stretch_InRelease
/var/lib/apt/lists/packagecloud.io_netdata_netdata_debian_dists_stretch_Release.gpg
/var/lib/apt/lists/packagecloud.io_netdata_netdata_debian_dists_stretch_Release
/var/lib/apt/lists/packagecloud.io_netdata_netdata-edge_debian_dists_stretch_main_binary-amd64_Packages
/var/lib/apt/lists/packagecloud.io_netdata_netdata-repoconfig_debian_dists_stretch_InRelease
/var/lib/apt/lists/packagecloud.io_netdata_netdata_debian_dists_stretch_main_binary-amd64_Packages
/var/lib/dpkg/info/netdata-repo-edge.list
/var/lib/dpkg/info/netdata-repo.list

Topic		Replies	Views
Borked installation and inability to re-install/un-install Help agent	6	774	August 24, 2023
Can't uninstall Netdata Help	1	225	September 21, 2024
After some struggles to update netdata agent, finally it got updated, but now the service fails to start. Help agent , installation , update	21	4561	September 29, 2022
Uninstall failed Help agent , installation	4	1731	January 3, 2023
netdata-kickstart.sh --reinstall-clean --static-only a few oddities Help agent , installation	10	1017	July 21, 2023

Tried uninstalling netdata, it failed

Related topics