Automatic netdata Updater fails

Help
,
1

Hi, I’m deeply impressed from this fine tool.

I did this ask @discord, without an reply, Anyway, posting with code tags is more clear. :slight_smile:

Environment

Debian Buster native installation. installed netdata with bash <(curl -Ss https://my-netdata.io/kickstart-static64.sh)

It works so far, but Updates are failing.

Problem/Question

Updates fails with an Error “Permission denied” every time

# bash <(curl -Ss https://my-netdata.io/kickstart-static64.sh)
 --- Found existing install of Netdata under: /opt/netdata --- 
 --- Attempting to update existing install instead of creating a new one --- 
[/var/lib/fail2ban]# /opt/netdata/usr/libexec/netdata/netdata-updater.sh --not-running-from-cron 
Mo 17. Jan 10:38:29 CET 2022 : INFO:  Checking if a newer version of the updater script is available.
/opt/netdata/usr/libexec/netdata/netdata-updater.sh: line 120: /tmp/netdata-test.pcBiIZOo7D: Permission denied
/opt/netdata/usr/libexec/netdata/netdata-updater.sh: line 120: /tmp/netdata-test.M5DBi1vyIm: Permission denied
Mo 17. Jan 10:38:30 CET 2022 : INFO:  Entering /tmp/netdata-updater-0RZujQJrGk
Mo 17. Jan 10:38:30 CET 2022 : INFO:  Current Version: 00103200100071
Mo 17. Jan 10:38:30 CET 2022 : INFO:  Latest Version: 00103200100078
Mo 17. Jan 10:38:30 CET 2022 : INFO:  Update available
Mo 17. Jan 10:38:30 CET 2022 : ERROR:  FAILED TO UPDATE NETDATA : Static binary checksum validation failed. Stopping netdata installation and leaving binary in /tmp/netdata-updater-0RZujQJrGk\nUsually this is a result of an older copy of the file being cached somewhere and can be resolved by simply retrying in an hour.
 FAILED  

 ABORTED  Failed to update existing Netdata install

What I expected to happen

Can anyone lead me to the solution? What can I do?
Regards

2

Hi, @Bernd.

Was it once? This looks similar to Installation fails with "Tarball checksum validation failed" due to geoblock · Issue #11939 · netdata/netdata · GitHub (a temporary issue we couldn’t debug so far).

I’d suggest re-running the updater manually (or waiting until the next cron run) and checking if the issue is gone or not.

3

No, the issue persists since some days. I did test it round about 10 times

4

Have you tried to run the updater script manually?

/opt/netdata/usr/libexec/netdata/netdata-updater.sh

5

Yes several times, always the error @line 120 appears

6

I think I’ve found the problem - fixed in fix(updater): checksum validation for static build by ilyam8 · Pull Request #11986 · netdata/netdata · GitHub.

Updating should start working fine after the fix is merged.

7

Now I get this it is getting better

/opt/netdata/usr/libexec/netdata/netdata-updater.sh
Mon 17 Jan 2022 07:24:54 PM CET : INFO:  Checking if a newer version of the updater script is available.
Mon 17 Jan 2022 07:24:55 PM CET : INFO:  Downloading newest version of updater script.
/opt/netdata/usr/libexec/netdata/netdata-updater.sh: line 120: /tmp/netdata-test.asd1ZONNKe: Permission denied
/opt/netdata/usr/libexec/netdata/netdata-updater.sh: line 120: /tmp/netdata-test.eZZX7iWTkO: Permission denied
/opt/netdata/usr/libexec/netdata/netdata-updater.sh: line 233: /tmp/netdata-updater-Oz8emSkRcb/netdata-updater.sh: Permission denied

# ll /tmp/netdata-updater-Oz8emSkRcb/netdata-updater.sh 
-rwxr-xr-x 1 root root 21012 Jan 17 19:24 /tmp/netdata-updater-Oz8emSkRcb/netdata-updater.sh

Ok, My problem is, I do have the mount option noexec for /tmp
Must be omitted,
I did mount /tmp/ -o remount
then a second try was successul

As a newbie to netdata, I would say build the updater to be called using sh, then there is no harm if someone has mounted their /tmp with noxec, or use a different path…

But otherwise great feature, I’m thrilled.
Thanks for your time and thanks for netdata.
Bernd

8

The updater itself should work fine if invoked with a plain, POSIX-compliant shell, and it should be handling temporary directories correctly (it should try $TMPDIR, /tmp, and then the current working directory, in that order, searching for the first that is both writable and allows execution). It looks like you may have stumbled across a bug in the updater itself here, independent of the checksum validation issue for static builds.

9

Hi,
the Checksum error is here, again

mount /tmp -o remount,exec
/opt/netdata/usr/libexec/netdata/netdata-updater.sh
Mi 19. Jan 09:36:51 CET 2022 : INFO:  Checking if a newer version of the updater script is available.
Mi 19. Jan 09:36:51 CET 2022 : INFO:  Entering /tmp/netdata-updater-XB48GRLcP1
Mi 19. Jan 09:36:51 CET 2022 : INFO:  Current Version: 00103200100078
Mi 19. Jan 09:36:51 CET 2022 : INFO:  Latest Version: 00103200100096
Mi 19. Jan 09:36:51 CET 2022 : INFO:  Update available
Mi 19. Jan 09:36:52 CET 2022 : ERROR:  FAILED TO UPDATE NETDATA : Static binary checksum validation failed. Stopping netdata installation and leaving binary in /tmp/netdata-updater-XB48GRLcP1\nUsually this is a result of an older copy of the file being cached somewhere and can be resolved by simply retrying i

:thinking:

10

Hm, could be your “netdata-updater.sh” doesn’t contain the fix and the script self auto-update doesn’t work for some reason.

Mi 19. Jan 09:36:51 CET 2022 : INFO: Checking if a newer version of the updater script is available.
Mi 19. Jan 09:36:51 CET 2022 : INFO: Entering /tmp/netdata-updater-XB48GRLcP1

I see no “Downloading newest version of updater script” log line.

Mi 19. Jan 09:36:51 CET 2022 : INFO: Current Version: 00103200100078

The fix is 82nd (since the latest tag) commit:

[ilyam@pc netdata]$ git log --pretty="%s (%an)" $(git describe --tags --abbrev=0)..HEAD --reverse | awk '{print "["NR"]\t" $s}' | grep "checksum va"
[82]	fix(updater): checksum validation for static build (#11986) (Ilya Mashchenko)

Try

touch -t 1970-01-01T00:00:00 /opt/netdata/usr/libexec/netdata/netdata-updater.sh

And then re-run the updater script.

11

touch -t “197001011800” /opt/netdata/usr/libexec/netdata/netdata-updater.sh#Did the trick. Thank you :slight_smile:

12

I can confirm the autoupdate worked fine last night.
:ok_hand: