I have a netdata agent which doesn’t seem to be able to connect to the cloud despite already being claimed.
It was working fine the other week but the machine has been off a bit recently and so I booted it up, installed the latest ubuntu patches and rebooted.
Local web UI looks fine but warns me that I need to re-claim the node to use it with the cloud. The error log shows this at start-up:
2023-04-24 18:15:14: netdata INFO : ACLK_MAIN : HTTPS "GET" request to "app.netdata.cloud" finished with HTTP code: 200
2023-04-24 18:15:14: netdata INFO : ACLK_MAIN : Getting Cloud /env successful
2023-04-24 18:15:14: netdata INFO : ACLK_MAIN : New ACLK protobuf protocol negotiated successfully (/env response).
2023-04-24 18:15:14: netdata INFO : ANALYTICS : thread created with task id 25187
2023-04-24 18:15:14: netdata INFO : ANALYTICS : set name of thread 25187 to ANALYTICS
2023-04-24 18:15:14: netdata INFO : P[proc] : Using now_boottime_usec() for uptime (dt is 7 ms)
2023-04-24 18:15:14: netdata INFO : ACLK_MAIN : HTTPS "GET" request to "api.netdata.cloud" finished with HTTP code: 409
2023-04-24 18:15:14: netdata ERROR : ACLK_MAIN : ACLK_OTP Challenge HTTP code not 200 OK (got 409)
2023-04-24 18:15:14: netdata ERROR : ACLK_MAIN : Cloud returned EC="6B9cGe4E6t-1654495", Msg-Key:"ErrAgentAlreadyConnected", Msg:"delay retry 5m0s: agent already connected", BlockRetry:false, Backoff:300s (-1 unset by cloud) (errno 2, No such file or directory)
2023-04-24 18:15:14: netdata ERROR : ACLK_MAIN : Error getting challenge
2023-04-24 18:15:14: netdata ERROR : ACLK_MAIN : Error passing Challenge/Response to get OTP
2023-04-24 18:15:14: netdata INFO : ACLK_MAIN : Wait before attempting to reconnect in 300.000 seconds
2023-04-24 18:15:23: netdata INFO : PD[ioping] : thread with task id 25095 finished
2023-04-24 18:15:23: netdata INFO : PD[charts.d] : PLUGINSD: 'host:laptop-linux' connected to '/usr/libexec/netdata/plugins.d/charts.d.plugin' running on pid 25430
2023-04-24 18:15:23: netdata INFO : PD[charts.d] : PLUGINSD: plugin called EXIT.
2023-04-24 18:15:23: netdata INFO : PD[charts.d] : PLUGINSD: 'host:laptop-linux', '/usr/libexec/netdata/plugins.d/charts.d.plugin' (pid 25430) disconnected after 0 successful data collections (ENDs).
2023-04-24 18:15:23: netdata INFO : PD[charts.d] : PLUGINSD: 'host:laptop-linux', '/usr/libexec/netdata/plugins.d/charts.d.plugin' (pid 25430) does not generate useful output but it reports success (exits with 0). Waiting a bit before starting it again..
2023-04-24 18:15:24: netdata ERROR : PLUGIN[cgroups] : child pid 25471 exited with code 1.
Before I try to fix it do you guys want anything?
OS: Ubuntu 22.10 on amd64.
Netdata: v1.38.0-421-nightly (installed via kickstarter script):
Version: netdata v1.38.0-421-nightly
Configure options: '--build=x86_64-linux-gnu' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--disable-option-checking' '--disable-silent-rules' '--libdir=${prefix}/lib/x86_64-linux-gnu' '--libexecdir=${prefix}/lib/x86_64-linux-gnu' '--disable-maintainer-mode' '--prefix=/usr' '--sysconfdir=/etc' '--localstatedir=/var' '--libdir=/usr/lib' '--libexecdir=/usr/libexec' '--with-user=netdata' '--with-math' '--with-zlib' '--with-webdir=/var/lib/netdata/www' '--disable-dependency-tracking' 'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -ffile-prefix-map=/usr/src/netdata=. -fstack-protector-strong -Wformat -Werror=format-security' 'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -ffile-prefix-map=/usr/src/netdata=. -fstack-protector-strong -Wformat -Werror=format-security'
Install type: binpkg-deb
Binary architecture: x86_64
Packaging distro:
Features:
dbengine: YES
Native HTTPS: YES
Netdata Cloud: YES
ACLK: YES
TLS Host Verification: YES
Machine Learning: YES
Stream Compression: YES
Libraries:
protobuf: YES (system)
jemalloc: NO
JSON-C: YES
libcap: NO
libcrypto: YES
libm: YES
tcalloc: NO
zlib: YES
Plugins:
apps: YES
cgroup Network Tracking: YES
CUPS: YES
EBPF: YES
IPMI: YES
NFACCT: YES
perf: YES
slabinfo: YES
Xen: NO
Xen VBD Error Tracking: NO
Exporters:
AWS Kinesis: NO
GCP PubSub: NO
MongoDB: NO
Prometheus Remote Write: YES
Debug/Developer Features:
Trace Allocations: NO
Luis.