Netdata broke yet again after auto update on 2 nodes

Help
1

Jun 16 07:14:45 ns1001839 systemd[1]: netdata.service: Main process exited, code=dumped, status=11/SEGV
Jun 16 07:14:45 ns1001839 systemd[1]: netdata.service: Failed with result ‘core-dump’.
Jun 16 07:15:44 ns1001839 kernel: [13537805.356419] netdata[349781]: segfault at 2d0 ip 00007f0297769c5d sp 00007ffe086bd7e0 error 4 in libc.so.6[7f02976ff000+195000]
Jun 16 07:15:44 ns1001839 systemd[1]: netdata.service: Main process exited, code=dumped, status=11/SEGV
Jun 16 07:15:44 ns1001839 systemd[1]: netdata.service: Failed with result ‘core-dump’.
Jun 16 07:16:14 ns1001839 systemd[1]: netdata.service: Scheduled restart job, restart counter is at 56.

2

Same thing happens after full delete of netdata and then install again…

3 
2023-06-16 07:20:21: netdata INFO  : MAIN : NETDATA_INSTANCE_CLOUD_TYPE=unknown
2023-06-16 07:20:21: netdata INFO  : MAIN : NETDATA_INSTANCE_CLOUD_INSTANCE_TYPE=unknown
2023-06-16 07:20:21: netdata INFO  : MAIN : NETDATA_INSTANCE_CLOUD_INSTANCE_REGION=unknown
2023-06-16 07:20:21: netdata INFO  : MAIN : NETDATA STARTUP: in     205 ms, collecting system info - next: initialize RRD structures
2023-06-16 07:20:21: netdata INFO  : MAIN : SQLite database /var/cache/netdata/netdata-meta.db initialization
2023-06-16 07:20:21: netdata INFO  : MAIN : Database version is 0, current version is 8. Running migration for metadata ...
2023-06-16 07:20:21: netdata INFO  : MAIN : Running database migration v0 to v1
2023-06-16 07:20:21: netdata INFO  : MAIN : Running "v1 to v2" database migration
2023-06-16 07:20:21: netdata INFO  : MAIN : Running "v2 to v3" database migration
2023-06-16 07:20:21: netdata INFO  : MAIN : Running database migration v3 to v4
2023-06-16 07:20:21: netdata INFO  : MAIN : Running "v4 to v5" database migration
2023-06-16 07:20:21: netdata INFO  : MAIN : Running "v5 to v6" database migration
2023-06-16 07:20:21: netdata INFO  : MAIN : Running "v6 to v7" database migration
2023-06-16 07:20:21: netdata INFO  : MAIN : Running database migration v7 to v8
2023-06-16 07:20:21: netdata ERROR : MAIN : SQLite error during database setup, rc = 14 (unable to open database file)
2023-06-16 07:20:21: netdata ERROR : MAIN : SQLite failed statement PRAGMA user_version=8;
2023-06-16 07:20:21: netdata FATAL : netdata : Failed to initialize SQLite
2023-06-16 07:20:21: netdata INFO  : MAIN : /usr/libexec/netdata/plugins.d/anonymous-statistics.sh 'FATAL' 'netdata:netdata' '0964@database/r:rrd_init       /0'
2023-06-16 07:20:23: netdata ERROR : MAIN : child pid 363790 exited with code 28.
2023-06-16 07:20:23: netdata ERROR : MAIN : Execution of anonymous statistics script returned 28.
2023-06-16 07:20:23: netdata ERROR : MAIN : Execution of anonymous statistics script returned http code 000.
/usr/sbin/netdata(+0x9990b)[0x5638eda3290b]
/usr/sbin/netdata(+0x1f3f75)[0x5638edb8cf75]
/usr/sbin/netdata(+0x54d79)[0x5638ed9edd79]
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f3e31ecbd90]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f3e31ecbe40]
/usr/sbin/netdata(+0x56e35)[0x5638ed9efe35]
2023-06-16 07:20:23: netdata INFO  : MAIN : NETDATA SHUTDOWN: initializing shutdown with code 1...
2023-06-16 07:20:23: netdata INFO  : MAIN : /usr/libexec/netdata/plugins.d/anonymous-statistics.sh 'EXIT' 'ERROR' '-'
2023-06-16 07:20:26: netdata ERROR : MAIN : child pid 363817 exited with code 28.
2023-06-16 07:20:26: netdata ERROR : MAIN : Execution of anonymous statistics script returned 28.
2023-06-16 07:20:26: netdata ERROR : MAIN : Execution of anonymous statistics script returned http code 000.
2023-06-16 07:20:26: netdata INFO  : MAIN : NETDATA SHUTDOWN: next: create shutdown file
2023-06-16 07:20:26: netdata INFO  : MAIN : NETDATA SHUTDOWN: in       0 ms, create shutdown file - next: close webrtc connections
2023-06-16 07:20:26: netdata INFO  : MAIN : NETDATA SHUTDOWN: in       0 ms, close webrtc connections - next: disable ML detection and training threads
4

After uninstall had to do
rm -rf /var/cache/netdata
rm -rf /usr/libexec/netdata

5

Auto updates are great but they sometimes break netdata. I’m not sure if I should revert to manual updates.
Also it’s possible I got this issue is because I’ve used nightly. Now I will use stable.

6

Which exact version of the Agent was this? We have an incident open for the latest nightly: Netdata Status - Agent connectivity problem

7

I was rushing to fix this I haven’t considered looking into version. Let’s close this post. I’ve changed to use stable :smiley:

1 Like