I am having trouble monitoring a node in Netdata cloud, my node appears unreachable

If I access the IP address of the node through the Netdata port the panel appears perfectly and monitors successfully, but, when I access from Netdata cloud (https://app.netdata.cloud), the node appears unreachable.

My setup is Linux (Ubuntu Server 18.04 LTS)
The error that was displayed when I ran the installation script (kickstart) is “already claimed”
So I run the claim script to claim the node
sudo ./[netdata-claim.sh](http://netdata-claim.sh/) -token=<Token> -id=$(uuidgen) -rooms=<Room ID> -url=[https://app.netdata.cloud](https://app.netdata.cloud/)

It seems to work locally, in the panel now appears the node, but, it appears as unreachable on the Cloud.

Is it possible to get netdata -W buildinfo , netdatacli aclk-state and grep -i ACLK error.log

netdata -W buildinfo
Version: netdata v1.32.1-30-nightly
Configure options: ‘–prefix=/usr’ ‘–sysconfdir=/etc’ ‘–localstatedir=/var’ ‘–libexecdir=/usr/libexec’ ‘–libdir=/usr/lib’ ‘–with-zlib’ ‘–with-math’ ‘–with-user=netdata’ ‘–with-bundled-lws’ ‘–with-bundled-protobuf’ ‘CFLAGS=-O2’ ‘LDFLAGS=’
Features:
dbengine: YES
Native HTTPS: YES
Netdata Cloud: YES
ACLK Next Generation: YES
ACLK-NG New Cloud Protocol: YES
ACLK Legacy: YES
TLS Host Verification: YES
Machine Learning: YES
Libraries:
protobuf: YES (bundled)
jemalloc: NO
JSON-C: YES
libcap: NO
libcrypto: YES
libm: YES
LWS: YES static v3.2.2
mosquitto: YES
tcalloc: NO
zlib: YES
Plugins:
apps: YES
cgroup Network Tracking: YES
CUPS: NO
EBPF: YES
IPMI: NO
NFACCT: NO
perf: YES
slabinfo: YES
Xen: NO
Xen VBD Error Tracking: NO
Exporters:
AWS Kinesis: NO
GCP PubSub: NO
MongoDB: NO
Prometheus Remote Write: NO

netdatacli aclk-state
ACLK Available: Yes
ACLK Implementation: Next Generation
New Cloud Protocol Support: Yes
Claimed: Yes
Claimed Id: 697c2a92-b4e0-46f9-8c26-129ea256b2c3
Online: Yes
Used Cloud Protocol: Legacy

The last thing you asked me, I don’t know how to give it to you. If you indicate the command in detail, I will gladly provide you with the requested information.

@iLxlo Thank you
Try grep -i ACLK /var/log/netdata/error.log.

From the netdatacli aclk-state output it seems agent believes it is connected to cloud. @papazach can we search this agent on VerneMQ by claim id above?

grep -i ACLK /var/log/netdata/error.log
2021-12-28 15:36:14: netdata INFO : MAIN : SQLite aclk sync initialization
2021-12-28 15:36:14: netdata INFO : MAIN : SQLite aclk sync initialization completed
2021-12-28 15:36:15: netdata INFO : ACLK_Main : thread created with task id 26386
2021-12-28 15:36:15: netdata INFO : ACLK_Main : set name of thread 26386 to ACLK_Main
2021-12-28 15:36:15: netdata INFO : ACLK_Main : Starting ACLK-NG
2021-12-28 15:36:15: netdata INFO : ACLK_Main : Waiting for Cloud to be enabled
2021-12-28 15:36:20: netdata INFO : ACLK_Main : Wait before attempting to reconnect in 0.000 seconds
2021-12-28 15:36:20: netdata INFO : ACLK_Main : Attempting connection now
2021-12-28 15:36:20: netdata INFO : ACLK_Stats : thread created with task id 26756
2021-12-28 15:36:20: netdata INFO : ACLK_Stats : set name of thread 26756 to ACLK_Stats
2021-12-28 15:36:20: netdata INFO : ACLK_Main : HTTPS “GET” request to “app.netdata.cloud” finished with HTTP code: 200
2021-12-28 15:36:20: netdata INFO : ACLK_Main : Getting Cloud /env successful
2021-12-28 15:36:21: netdata INFO : ACLK_Main : HTTPS “GET” request to “app.netdata.cloud” finished with HTTP code: 200
2021-12-28 15:36:21: netdata INFO : ACLK_Main : ACLK_OTP Got Challenge from Cloud
2021-12-28 15:36:21: netdata INFO : ACLK_Main : HTTPS “POST” request to “app.netdata.cloud” finished with HTTP code: 201
2021-12-28 15:36:21: netdata INFO : ACLK_Main : ACLK_OTP Got Password from Cloud
2021-12-28 15:36:22: netdata INFO : ACLK_Main : [mqtt_wss] I: ws_client: Websocket Connection Accepted By Server
2021-12-28 15:36:22: netdata INFO : ACLK_Main : ACLK connection successfully established
2021-12-28 15:36:22: netdata ERROR : ACLK_Main : ACLK localhost popcorn timer - wait 2 seconds longer
2021-12-28 15:36:24: netdata ERROR : ACLK_Main : ACLK localhost popcorn timer - wait 3 seconds longer
2021-12-28 15:36:27: netdata ERROR : ACLK_Main : ACLK localhost popcorn timer - wait 2 seconds longer
2021-12-28 15:36:29: netdata ERROR : ACLK_Main : ACLK localhost popcorn timer finished
2021-12-28 15:36:29: netdata INFO : ACLK_Main : Starting 2 query threads.
2021-12-28 15:36:29: netdata INFO : ACLK_Query_0 : thread created with task id 26810
2021-12-28 15:36:29: netdata INFO : ACLK_Query_0 : set name of thread 26810 to ACLK_Query_0
2021-12-28 15:36:29: netdata INFO : ACLK_Query_1 : thread created with task id 26811
2021-12-28 15:36:29: netdata INFO : ACLK_Query_1 : set name of thread 26811 to ACLK_Query_1
2021-12-28 16:01:16: netdata ERROR : ACLK_Main : Connection Error or Dropped
2021-12-28 16:01:16: netdata INFO : ACLK_Main : Wait before attempting to reconnect in 0.000 seconds
2021-12-28 16:01:16: netdata INFO : ACLK_Main : Attempting connection now
2021-12-28 16:01:17: netdata INFO : ACLK_Main : HTTPS “GET” request to “app.netdata.cloud” finished with HTTP code: 200
2021-12-28 16:01:17: netdata INFO : ACLK_Main : Getting Cloud /env successful
2021-12-28 16:01:17: netdata INFO : ACLK_Main : HTTPS “GET” request to “app.netdata.cloud” finished with HTTP code: 200
2021-12-28 16:01:17: netdata INFO : ACLK_Main : ACLK_OTP Got Challenge from Cloud
2021-12-28 16:01:18: netdata INFO : ACLK_Main : HTTPS “POST” request to “app.netdata.cloud” finished with HTTP code: 201
2021-12-28 16:01:18: netdata INFO : ACLK_Main : ACLK_OTP Got Password from Cloud
2021-12-28 16:01:18: netdata INFO : ACLK_Main : [mqtt_wss] I: ws_client: Websocket Connection Accepted By Server
2021-12-28 16:01:18: netdata INFO : ACLK_Main : ACLK connection successfully established
2021-12-28 16:01:18: netdata ERROR : ACLK_Main : Sending connect payload immediately as popcorning was finished already.
2021-12-28 16:04:36: netdata INFO : MAIN : EXIT: Stopping main thread: ACLK_Main
2021-12-28 16:04:37: netdata INFO : ACLK_Main : Preparing to gracefully shutdown ACLK connection
2021-12-28 16:04:37: netdata INFO : ACLK_Query_0 : thread with task id 26810 finished
2021-12-28 16:04:37: netdata INFO : ACLK_Stats : thread with task id 26756 finished
2021-12-28 16:04:37: netdata INFO : ACLK_Main : Shutdown message has been acknowledged by the cloud. Exiting gracefully
2021-12-28 16:04:37: netdata INFO : ACLK_Main : MQTT App Layer disconnect message sent successfully
2021-12-28 16:04:37: netdata INFO : ACLK_Main : ACLK link is down
2021-12-28 16:04:37: netdata INFO : ACLK_Main : Attempting to gracefully shutdown the MQTT/WSS connection
2021-12-28 16:04:37: netdata INFO : ACLK_Main : [mqtt_wss] I: ws_client: WebSocket server closed the connection with EC=1000. Without message.
2021-12-28 16:04:37: netdata INFO : ACLK_Query_1 : thread with task id 26811 finished
2021-12-28 16:04:37: netdata INFO : ACLK_Main : thread with task id 26386 finished
2021-12-28 17:00:41: netdata INFO : MAIN : SQLite aclk sync initialization
2021-12-28 17:00:41: netdata INFO : MAIN : SQLite aclk sync initialization completed
2021-12-28 17:00:42: netdata INFO : ACLK_Main : thread created with task id 766
2021-12-28 17:00:42: netdata INFO : ACLK_Main : set name of thread 766 to ACLK_Main
2021-12-28 17:00:42: netdata INFO : ACLK_Main : Starting ACLK-NG
2021-12-28 17:00:42: netdata INFO : ACLK_Main : Waiting for Cloud to be enabled
2021-12-28 17:00:47: netdata INFO : ACLK_Main : Wait before attempting to reconnect in 0.000 seconds
2021-12-28 17:00:47: netdata INFO : ACLK_Main : Attempting connection now
2021-12-28 17:00:47: netdata INFO : ACLK_Stats : thread created with task id 1059
2021-12-28 17:00:47: netdata INFO : ACLK_Stats : set name of thread 1059 to ACLK_Stats
2021-12-28 17:00:48: netdata INFO : ACLK_Main : HTTPS “GET” request to “app.netdata.cloud” finished with HTTP code: 200
2021-12-28 17:00:48: netdata INFO : ACLK_Main : Getting Cloud /env successful
2021-12-28 17:00:48: netdata INFO : ACLK_Main : HTTPS “GET” request to “app.netdata.cloud” finished with HTTP code: 200
2021-12-28 17:00:48: netdata INFO : ACLK_Main : ACLK_OTP Got Challenge from Cloud
2021-12-28 17:00:49: netdata INFO : ACLK_Main : HTTPS “POST” request to “app.netdata.cloud” finished with HTTP code: 201
2021-12-28 17:00:49: netdata INFO : ACLK_Main : ACLK_OTP Got Password from Cloud
2021-12-28 17:00:49: netdata INFO : ACLK_Main : [mqtt_wss] I: ws_client: Websocket Connection Accepted By Server
2021-12-28 20:11:33: netdata INFO : MAIN : COMMAND: Reopening aclk/cloud state.
2021-12-28 20:15:21: netdata INFO : MAIN : EXIT: Stopping main thread: ACLK_Main
2021-12-28 20:15:21: netdata INFO : ACLK_Main : Preparing to gracefully shutdown ACLK connection
2021-12-28 20:15:21: netdata INFO : ACLK_Query_1 : thread with task id 1108 finished
2021-12-28 20:15:21: netdata INFO : ACLK_Main : Shutdown message has been acknowledged by the cloud. Exiting gracefully
2021-12-28 20:15:21: netdata INFO : ACLK_Main : MQTT App Layer disconnect message sent successfully
2021-12-28 20:15:21: netdata INFO : ACLK_Main : ACLK link is down
2021-12-28 20:15:21: netdata INFO : ACLK_Main : Attempting to gracefully shutdown the MQTT/WSS connection
2021-12-28 20:15:21: netdata INFO : ACLK_Main : [mqtt_wss] I: ws_client: WebSocket server closed the connection with EC=1000. Without message.
2021-12-28 20:15:21: netdata INFO : ACLK_Query_0 : thread with task id 1107 finished
2021-12-28 20:15:22: netdata INFO : ACLK_Stats : thread with task id 1059 finished
2021-12-28 20:15:22: netdata INFO : ACLK_Main : thread with task id 766 finished
2021-12-29 12:14:30: netdata INFO : MAIN : SQLite aclk sync initialization
2021-12-29 12:14:30: netdata INFO : MAIN : SQLite aclk sync initialization completed
2021-12-29 12:14:30: netdata INFO : ACLK_Main : thread created with task id 804
2021-12-29 12:14:30: netdata INFO : ACLK_Main : set name of thread 804 to ACLK_Main
2021-12-29 12:14:30: netdata INFO : ACLK_Main : Starting ACLK-NG
2021-12-29 12:14:30: netdata INFO : ACLK_Main : Waiting for Cloud to be enabled
2021-12-29 12:14:36: netdata INFO : ACLK_Main : Wait before attempting to reconnect in 0.000 seconds
2021-12-29 12:14:36: netdata INFO : ACLK_Main : Attempting connection now
2021-12-29 12:14:36: netdata INFO : ACLK_Stats : thread created with task id 1147
2021-12-29 12:14:36: netdata INFO : ACLK_Stats : set name of thread 1147 to ACLK_Stats
2021-12-29 12:14:37: netdata INFO : ACLK_Main : HTTPS “GET” request to “app.netdata.cloud” finished with HTTP code: 200
2021-12-29 12:14:37: netdata INFO : ACLK_Main : Getting Cloud /env successful
2021-12-29 12:14:37: netdata INFO : ACLK_Main : HTTPS “GET” request to “app.netdata.cloud” finished with HTTP code: 200
2021-12-29 12:14:37: netdata INFO : ACLK_Main : ACLK_OTP Got Challenge from Cloud
2021-12-29 12:14:38: netdata INFO : ACLK_Main : HTTPS “POST” request to “app.netdata.cloud” finished with HTTP code: 201
2021-12-29 12:14:38: netdata INFO : ACLK_Main : ACLK_OTP Got Password from Cloud
2021-12-29 12:14:38: netdata INFO : ACLK_Main : [mqtt_wss] I: ws_client: Websocket Connection Accepted By Server
2021-12-29 12:14:38: netdata INFO : ACLK_Main : ACLK connection successfully established

@iLxlo thank you. From the log, it seems to me the agent is indeed successfully connected to the cloud. I am asking around our cloud devs to take a look (why it appears offline in the cloud dashboard).

1 Like

Hi @iLxlo !

The agent looks to be online from the cloud, however it does not appear to belong to any room. Could you check the nodes available for the room you want and try adding this one?

Instructions on that can be found here: War Rooms | Learn Netdata.

Kind regards,
Christos

Done. It seems that the problem was more derived from the nodes that appear in the list of available nodes.

Apparently I had duplicated the same node x10 times, 7 of them were listed as “Available” (they did not have the unreachable mark). I have added all the nodes that were available and finally I have found the node that belongs to me.

The remainder were assigned the “unreachable” mark.

Thanks for your help and now you can close this thread!

2 Likes

Really happy to hear you now have access to your node @iLxlo !
Let us know if you need any further help and we’ll be happy to assist!

1 Like