Periodically Anacron job 'cron.daily' on localhost reports a problem with netdata-updater.sh

Help
1

Not actually an agent-health issue but that’s the closest tag. This is Ubuntu 20.04.1LTS, anacron runs a daily job starting at 6:25 - 25 6 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ). Netdata-updater will run at around 7:35. I have seen two reports about update issues as cron messages. Received a third one this morning so thought I’d bring it to someone’s attention. The first report was on 11/18 this year:

/etc/cron.daily/netdata-updater:

/etc/cron.daily/netdata-updater: line 266: THIS_SHOULD_BE_REPLACED_BY_INSTALLER_SCRIPT: No such file or directory run-parts: /etc/cron.daily/netdata-updater exited with return code 1

The next one was on 12/01 with the same output as above. The third was this morning with a different output:

/etc/cron.daily/netdata-updater:

/etc/cron.daily/netdata-updater: line 31: /etc/cron.daily/netdata-updater.sh: No such file or directory
run-parts: /etc/cron.daily/netdata-updater exited with return code 127

Line 31 in the updater script is:
script_source=“$(”$(CDPATH=‘’ cd – “$(dirname – “$0”)” && pwd -P)/netdata-updater.sh")"

Until I refreshed my netdata tab it showed no update was available at the same time my version was shown as v1.26.0-333-nightly while it showed v1.26.0-354-nightly was available. After I refreshed the tab the ‘Update Now’ showed up. I don’t know if *-354 was last night’s nightly or tonight’s nightly update.

This is not an everyday occurrence, only happened 3 times so far as shown above. Also, if I understand correctly each nightly update places a newer version of files in /usr/libexec/netdata. Looking in that directory I see that the files are still there from the 8th.

drwxr-xr-x 2 root root 4096 Dec 8 08:35 charts.d
-rwxr-x— 1 root root 12057 Dec 8 08:36 netdata-uninstaller.sh
-rwxr-xr-x 1 root root 12529 Dec 8 08:36 netdata-updater.sh
drwxr-xr-x 3 root root 4096 Dec 8 08:35 node.d
drwxr-xr-x 2 root root 4096 Dec 8 08:35 plugins.d
drwxr-xr-x 3 root root 4096 Dec 8 08:35 python.d

Trying to update just now I enter:

chris@localhost:~$ bash <(curl -Ss https://my-netdata.io/kickstart.sh)
— Found existing install of Netdata under: / —
— Attempting to update existing install instead of creating a new one —
[/home/chris]$ sudo //usr/libexec/netdata/netdata-updater.sh --not-running-from-cron [sudo] password for chris:

after entering my password it just sits there until I ctrl-c.

1 Like
2

This is a known bug in the updater script (ironically introduced by some code that was supposed to make it easier to recover from bugs in the updater script).

As of 2020-12-09, this is fixed in the nightlies.

Affected users will need to manually update the system using your original install method instead of the updater (make sure to add --reinstall and the same options you installed with originally if using the kickstart.sh or kickstart-static64.sh scripts, otherwise it will try to run the updater and run into a different issue caused by the same bug).

3

Thanks @Austin_Hemmelgarn, worked perfectly. Appreciate the quick reply.

4

Hey @chris.pollock,

If the response of Austin solved your question, please mark it as a solution, so other can see it!

Cheers

5

I ran the kickstart script with --reinstall on all my netdata hosts, and now each is reporting the same error via cron:

/etc/cron.daily/netdata-updater:
/etc/cron.daily/netdata-updater: line 101: _cannot_find_tmpdir: command not found

Should I do a full uninstall then reinstall?

6

@haydenseitz That’s an unrelated bug in the updater itself that should be fixed in the near future (and the fix should need no intervention on your part, as the updater now defaults to updating itself before updating Netdata). However, it should have zero impact on actual usability of the updater (it should function normally, it’s just the new feature that introduced the bug won’t work correctly).

2 Likes