I have a docker-based Netdata agent container running in Azure container instances based on a custom docker image that uses
FROM netdata/netdata:latest, installs some dependencies and sets up the configuration files.
I also use it as a statsd server for custom metrics I’m sending to it. It’s connected to Netdata cloud and sometimes I can access the metrics on app.netdata.cloud. However, often the node is shown as “offline” and the metrics are inaccessible; even though I can connect to the instance via SSH and the logs of the Netdata agent container look normal (it seems to be online).
- tried restarting the agent (after which it is mostly online for a while, but eventually it goes back offline)
- change different settings in netdata.conf - problem persisted for all combinations I tried
- the docker container only exposes a UDP port for the statsd server - 19999 is not exposed, since we only use Netdata cloud
- netdata agent version (netdata -v): netdata v1.36.0-333-nightly
- netdatacli aclk-state (shows “online yes”, but the node is not shown as online in the cloud dashboard):
ACLK Available: Yes ACLK Version: 2 Protocols Supported: Protobuf Protocol Used: Protobuf MQTT Version: 5 Claimed: Yes Claimed Id: xxxx Cloud URL: https://app.netdata.cloud Online: Yes Reconnect count: 4 Banned By Cloud: No Last Connection Time: 2022-11-10 15:22:58 Last Connection Time + 3 PUBACKs received: 2022-11-10 15:22:58 Last Disconnect Time: 2022-11-10 15:22:56 Received Cloud MQTT Messages: 20 MQTT Messages Confirmed by Remote Broker (PUBACKs): 19 > Node Instance for mGUID: "xxxx" hostname "SandboxHost-xxxx" Claimed ID: xxxxx Node ID: xxxxx Streaming Hops: 0 Relationship: self Alert Streaming Status: Updates: 1 Batch ID: 1 Last Acked Seq ID: 331 Pending Min Seq ID: 0 Pending Max Seq ID: 0 Last Submitted Seq ID: 331
The node is shown as online and the data is accessible in Netdata cloud whenever it is actually online (i.e., continuously).