Most our hosts can stream to the cloud

Help
1

In the cloud dashboard, most our hosts are shown with the status “unreachable”.

Looking into the logs on the hosts, it shows something like this:

2022-02-02 15:48:58: netdata INFO  : ACLK_Main : Attempting connection now
2022-02-02 15:48:58: netdata INFO  : ACLK_Main : HTTPS "GET" request to "app.netdata.cloud" finished with HTTP code: 200
2022-02-02 15:48:58: netdata INFO  : ACLK_Main : Getting Cloud /env successful
2022-02-02 15:48:58: netdata INFO  : ACLK_Main : Switching ACLK to new protobuf protocol. Due to /env response.
2022-02-02 15:48:59: netdata INFO  : ACLK_Main : HTTPS "GET" request to "app.netdata.cloud" finished with HTTP code: 200
2022-02-02 15:48:59: netdata INFO  : ACLK_Main : ACLK_OTP Got Challenge from Cloud
2022-02-02 15:48:59: netdata INFO  : ACLK_Main : HTTPS "POST" request to "app.netdata.cloud" finished with HTTP code: 201
2022-02-02 15:48:59: netdata INFO  : ACLK_Main : ACLK_OTP Got Password from Cloud
2022-02-02 15:49:00: netdata INFO  : ACLK_Main : [mqtt_wss] I: ws_client: Websocket Connection Accepted By Server
2022-02-02 15:49:00: netdata INFO  : ACLK_Main : ACLK connection successfully established
2022-02-02 15:49:00: netdata INFO  : ACLK_Main : Preparing to gracefully shutdown ACLK connection
2022-02-02 15:49:00: netdata ERROR : ACLK_Main : POP Query Queue is blocked from accepting new requests. This is normally the case when ACLK prepares to shutdown.
2022-02-02 15:49:00: netdata ERROR : MAIN : Query Queue is blocked from accepting new requests. This is normally the case when ACLK prepares to shutdown.
2022-02-02 15:49:00: netdata INFO  : ACLK_Main : Shutdown message has been acknowledged by the cloud. Exiting gracefully
2022-02-02 15:49:00: netdata INFO  : ACLK_Main : MQTT App Layer `disconnect` message sent successfully
2022-02-02 15:49:00: netdata INFO  : ACLK_Main : ACLK link is down
2022-02-02 15:49:00: netdata INFO  : ACLK_Main : Attempting to gracefully shutdown the MQTT/WSS connection
2022-02-02 15:49:00: netdata ERROR : ACLK_Main : Link is shutting down. Ignoring incoming message.
2022-02-02 15:49:00: netdata INFO  : ACLK_Main : [mqtt_wss] I: ws_client: WebSocket server closed the connection with EC=1000. Without message.
2022-02-02 15:49:00: netdata INFO  : ACLK_Main : Wait before attempting to reconnect in 261.864 seconds

When going to the host dashboard, all graphs are shown correctly. Just not the cloud dashboard.

For one of the hosts I already tried to reclaim an ID, makes no difference.

Please help.

2

I think I found the issue.
The agent will stay in an endless reconnect loop if it is reclaimed during runtime. I made a small PR to fix it in the next releases (incl. next nightly). For now however manual restart of netdata process is needed if agent has ever been reclaimed whilst running. :frowning:

3

That did the trick! Thanks a lot @underhood for this very quick turnaround.