Streaming error, disconnect / reconnect

Help
, ,
1

Hello,

I have a weird issue I can’t figure out, I hope someone will be able to help me.

I have around 20 servers monitored by Netdata, all servers are streaming to what I call a Netdata Master.
For 2 servers, charts are getting truncated after a while on the Netdata Master dashboard, while they are working as expected on the server itself.

Here is an example:

From Netdata Master:

Screenshot 2022-06-14 at 14.28.10
Screenshot 2022-06-14 at 14.28.101350×425 80.3 KB

From the Server itself:

Screenshot 2022-06-14 at 14.28.26
Screenshot 2022-06-14 at 14.28.261619×441 90.9 KB

From the Netdata Master, I can see the following logs:

2022-06-13 10:39:34: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:39518] : STREAM APACHE-PROD [XXX.XXX.XXX.XXX]:39518: receive thread created (task id 25254)
2022-06-13 10:39:34: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:39518] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:39518]: initializing communication...
2022-06-13 10:39:34: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:39518] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:39518]: Netdata is using the stream version 4.
2022-06-13 10:39:34: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:39518] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:39518]: receiving metrics...
2022-06-13 10:59:35: netdata ERROR : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:55688] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:55688]: disconnected (completed 176 updates).
2022-06-13 10:59:35: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:55688] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:55688]: receive thread ended (task id 29124)
2022-06-13 10:59:35: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:55704] : STREAM APACHE-PROD [XXX.XXX.XXX.XXX]:55704: receive thread created (task id 29127)
2022-06-13 10:59:35: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:55704] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:55704]: initializing communication...
2022-06-13 10:59:35: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:55704] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:55704]: Netdata is using the stream version 4.
2022-06-13 10:59:35: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:55704] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:55704]: receiving metrics...
2022-06-13 11:19:36: netdata ERROR : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:40196] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:40196]: disconnected (completed 113 updates).
2022-06-13 11:19:36: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:40196] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:40196]: receive thread ended (task id 580)
2022-06-13 11:19:36: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:40202] : STREAM APACHE-PROD [XXX.XXX.XXX.XXX]:40202: receive thread created (task id 583)
2022-06-13 11:19:36: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:40202] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:40202]: initializing communication...
2022-06-13 11:19:36: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:40202] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:40202]: Netdata is using the stream version 4.
2022-06-13 11:19:36: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:40202] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:40202]: receiving metrics...
2022-06-13 11:39:37: netdata ERROR : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:53534] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:53534]: disconnected (completed 154 updates).
2022-06-13 11:39:37: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:53534] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:53534]: receive thread ended (task id 4480)
2022-06-13 11:39:37: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:53546] : STREAM APACHE-PROD [XXX.XXX.XXX.XXX]:53546: receive thread created (task id 4483)
2022-06-13 11:39:37: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:53546] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:53546]: initializing communication...
2022-06-13 11:39:37: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:53546] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:53546]: Netdata is using the stream version 4.
2022-06-13 11:39:37: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:53546] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:53546]: receiving metrics...
2022-06-13 11:59:38: netdata ERROR : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:39410] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:39410]: disconnected (completed 215 updates).
2022-06-13 11:59:38: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:39410] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:39410]: receive thread ended (task id 8429)
2022-06-13 11:59:38: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:39418] : STREAM APACHE-PROD [XXX.XXX.XXX.XXX]:39418: receive thread created (task id 8432)
2022-06-13 11:59:38: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:39418] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:39418]: initializing communication...
2022-06-13 11:59:38: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:39418] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:39418]: Netdata is using the stream version 4.
2022-06-13 11:59:38: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:39418] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:39418]: receiving metrics...
2022-06-13 12:19:39: netdata ERROR : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:56348] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:56348]: disconnected (completed 308 updates).
2022-06-13 12:19:39: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:56348] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:56348]: receive thread ended (task id 12548)
2022-06-13 12:39:40: netdata ERROR : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:48652] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:48652]: disconnected (completed 343 updates).
2022-06-13 12:39:40: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:48652] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:48652]: receive thread ended (task id 16463)
2022-06-13 12:39:40: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:48678] : STREAM APACHE-PROD [XXX.XXX.XXX.XXX]:48678: receive thread created (task id 16466)
2022-06-13 12:39:40: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:48678] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:48678]: initializing communication...
2022-06-13 12:39:40: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:48678] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:48678]: Netdata is using the stream version 4.
2022-06-13 12:39:40: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:48678] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:48678]: receiving metrics...
2022-06-13 12:59:41: netdata ERROR : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:42100] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:42100]: disconnected (completed 242 updates).
2022-06-13 12:59:41: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:42100] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:42100]: receive thread ended (task id 20527)
2022-06-13 12:59:41: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:42134] : STREAM APACHE-PROD [XXX.XXX.XXX.XXX]:42134: receive thread created (task id 20530)
2022-06-13 12:59:41: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:42134] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:42134]: initializing communication...
2022-06-13 12:59:41: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:42134] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:42134]: Netdata is using the stream version 4.
2022-06-13 12:59:41: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:42134] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:42134]: receiving metrics...
2022-06-13 13:19:42: netdata ERROR : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:35804] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:35804]: disconnected (completed 350 updates).
2022-06-13 13:19:42: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:35804] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:35804]: receive thread ended (task id 24546)
2022-06-13 13:19:42: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:35818] : STREAM APACHE-PROD [XXX.XXX.XXX.XXX]:35818: receive thread created (task id 24549)
2022-06-13 13:19:42: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:35818] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:35818]: initializing communication...
2022-06-13 13:19:42: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:35818] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:35818]: Netdata is using the stream version 4.
2022-06-13 13:19:42: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:35818] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:35818]: receiving metrics...
2022-06-13 13:39:43: netdata ERROR : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:57882] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:57882]: disconnected (completed 172 updates).
2022-06-13 13:39:43: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:57882] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:57882]: receive thread ended (task id 28588)
2022-06-13 13:39:43: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:57886] : STREAM APACHE-PROD [XXX.XXX.XXX.XXX]:57886: receive thread created (task id 28611)
2022-06-13 13:39:43: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:57886] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:57886]: initializing communication...
2022-06-13 13:39:43: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:57886] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:57886]: Netdata is using the stream version 4.
2022-06-13 13:39:43: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:57886] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:57886]: receiving metrics...
2022-06-14 12:23:23: netdata ERROR : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:37172] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:37172]: disconnected (completed 108 updates).
2022-06-14 12:23:23: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:37172] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:37172]: receive thread ended (task id 20362)
2022-06-14 12:23:23: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:37186] : STREAM APACHE-PROD [XXX.XXX.XXX.XXX]:37186: receive thread created (task id 20365)
2022-06-14 12:23:23: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:37186] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:37186]: initializing communication...
2022-06-14 12:23:23: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:37186] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:37186]: Netdata is using the stream version 4.
2022-06-14 12:23:23: netdata INFO  : STREAM_RECEIVER[APACHE-PROD,[XXX.XXX.XXX.XXX]:37186] : STREAM APACHE-PROD [receive from [XXX.XXX.XXX.XXX]:37186]: receiving metrics...

From the server itself, logs are the following:

2022-06-14 12:18:27: netdata ERROR : PLUGINSD[nfacct] : STREAM APACHE-PROD [send]: not ready - discarding collected metrics.
2022-06-14 12:18:27: netdata INFO  : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send to YYY.YYY.YYY.YYY:19999]: initializing communication...
2022-06-14 12:18:27: netdata INFO  : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send to YYY.YYY.YYY.YYY:19999]: waiting response from remote netdata...
2022-06-14 12:18:27: netdata INFO  : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send to YYY.YYY.YYY.YYY:19999]: established communication with a parent using protocol version 4 - ready to send metrics...
2022-06-14 12:18:27: netdata ERROR : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send]: discarding 1796 bytes of metrics already in the buffer.
2022-06-14 12:18:27: netdata INFO  : PLUGIN[cgroups] : STREAM APACHE-PROD [send]: sending metrics...
2022-06-14 12:18:28: netdata ERROR : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send to YYY.YYY.YYY.YYY:19999]: failed to send metrics - closing connection - we have sent 288460 bytes on this connection. (errno 104, Connection reset by peer)
2022-06-14 12:18:28: netdata INFO  : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send to YYY.YYY.YYY.YYY:19999]: connecting...
2022-06-14 12:18:28: netdata ERROR : PLUGIN[proc] : STREAM APACHE-PROD [send]: not ready - discarding collected metrics. (errno 2, No such file or directory)
2022-06-14 12:18:28: netdata INFO  : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send to YYY.YYY.YYY.YYY:19999]: initializing communication...
2022-06-14 12:18:28: netdata INFO  : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send to YYY.YYY.YYY.YYY:19999]: waiting response from remote netdata...
2022-06-14 12:18:28: netdata INFO  : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send to YYY.YYY.YYY.YYY:19999]: established communication with a parent using protocol version 4 - ready to send metrics...
2022-06-14 12:18:28: netdata ERROR : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send]: discarding 302 bytes of metrics already in the buffer.
2022-06-14 12:18:28: netdata INFO  : PLUGIN[proc] : STREAM APACHE-PROD [send]: sending metrics...
2022-06-14 12:18:29: netdata ERROR : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send to YYY.YYY.YYY.YYY:19999]: failed to send metrics - closing connection - we have sent 214772 bytes on this connection. (errno 104, Connection reset by peer)
2022-06-14 12:18:29: netdata INFO  : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send to YYY.YYY.YYY.YYY:19999]: connecting...
2022-06-14 12:18:29: netdata ERROR : PLUGINSD[python.d] : STREAM APACHE-PROD [send]: not ready - discarding collected metrics.
2022-06-14 12:18:29: netdata INFO  : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send to YYY.YYY.YYY.YYY:19999]: initializing communication...
2022-06-14 12:18:29: netdata INFO  : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send to YYY.YYY.YYY.YYY:19999]: waiting response from remote netdata...
2022-06-14 12:18:29: netdata INFO  : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send to YYY.YYY.YYY.YYY:19999]: established communication with a parent using protocol version 4 - ready to send metrics...
2022-06-14 12:18:29: netdata ERROR : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send]: discarding 2003 bytes of metrics already in the buffer.
2022-06-14 12:18:29: netdata INFO  : PLUGIN[proc] : STREAM APACHE-PROD [send]: sending metrics...
2022-06-14 12:18:30: netdata ERROR : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send to YYY.YYY.YYY.YYY:19999]: error during read (-1). Restarting connection (errno 104, Connection reset by peer)
2022-06-14 12:18:30: netdata INFO  : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send to YYY.YYY.YYY.YYY:19999]: connecting...
2022-06-14 12:18:30: netdata ERROR : PLUGIN[proc] : STREAM APACHE-PROD [send]: not ready - discarding collected metrics. (errno 2, No such file or directory)
2022-06-14 12:18:30: netdata INFO  : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send to YYY.YYY.YYY.YYY:19999]: initializing communication...
2022-06-14 12:18:30: netdata INFO  : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send to YYY.YYY.YYY.YYY:19999]: waiting response from remote netdata...
2022-06-14 12:18:30: netdata INFO  : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send to YYY.YYY.YYY.YYY:19999]: established communication with a parent using protocol version 4 - ready to send metrics...
2022-06-14 12:18:30: netdata INFO  : PLUGINSD[ebpf] : STREAM APACHE-PROD [send]: sending metrics...
2022-06-14 12:18:31: netdata ERROR : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send to YYY.YYY.YYY.YYY:19999]: error during read (-1). Restarting connection (errno 104, Connection reset by peer)
2022-06-14 12:18:31: netdata INFO  : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send to YYY.YYY.YYY.YYY:19999]: connecting...
2022-06-14 12:18:31: netdata ERROR : PLUGINSD[python.d] : STREAM APACHE-PROD [send]: not ready - discarding collected metrics.
2022-06-14 12:18:31: netdata INFO  : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send to YYY.YYY.YYY.YYY:19999]: initializing communication...
2022-06-14 12:18:31: netdata INFO  : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send to YYY.YYY.YYY.YYY:19999]: waiting response from remote netdata...
2022-06-14 12:18:31: netdata INFO  : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send to YYY.YYY.YYY.YYY:19999]: established communication with a parent using protocol version 4 - ready to send metrics...
2022-06-14 12:18:31: netdata INFO  : PLUGIN[proc] : STREAM APACHE-PROD [send]: sending metrics...
2022-06-14 12:18:32: netdata ERROR : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send to YYY.YYY.YYY.YYY:19999]: error during read (-1). Restarting connection (errno 104, Connection reset by peer)
2022-06-14 12:18:32: netdata INFO  : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send to YYY.YYY.YYY.YYY:19999]: connecting...
2022-06-14 12:18:32: netdata INFO  : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send to YYY.YYY.YYY.YYY:19999]: initializing communication...
2022-06-14 12:18:32: netdata INFO  : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send to YYY.YYY.YYY.YYY:19999]: waiting response from remote netdata...
2022-06-14 12:18:32: netdata INFO  : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send to YYY.YYY.YYY.YYY:19999]: established communication with a parent using protocol version 4 - ready to send metrics...
2022-06-14 12:18:32: netdata ERROR : PLUGIN[cgroups] : STREAM APACHE-PROD [send]: not ready - discarding collected metrics.
2022-06-14 12:18:32: netdata INFO  : PLUGIN[cgroups] : STREAM APACHE-PROD [send]: sending metrics...
2022-06-14 12:18:33: netdata ERROR : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send to YYY.YYY.YYY.YYY:19999]: connection closed by far end. Restarting connection
2022-06-14 12:18:33: netdata ERROR : PLUGINSD[python.d] : STREAM APACHE-PROD [send]: not ready - discarding collected metrics.
2022-06-14 12:18:33: netdata INFO  : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send to YYY.YYY.YYY.YYY:19999]: connecting...
2022-06-14 12:18:33: netdata INFO  : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send to YYY.YYY.YYY.YYY:19999]: initializing communication...
2022-06-14 12:18:33: netdata INFO  : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send to YYY.YYY.YYY.YYY:19999]: waiting response from remote netdata...
2022-06-14 12:18:33: netdata INFO  : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send to YYY.YYY.YYY.YYY:19999]: established communication with a parent using protocol version 4 - ready to send metrics...
2022-06-14 12:18:33: netdata INFO  : PLUGIN[proc] : STREAM APACHE-PROD [send]: sending metrics...
2022-06-14 12:18:34: netdata ERROR : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send to YYY.YYY.YYY.YYY:19999]: failed to send metrics - closing connection - we have sent 286718 bytes on this connection. (errno 104, Connection reset by peer)
2022-06-14 12:18:34: netdata INFO  : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send to YYY.YYY.YYY.YYY:19999]: connecting...
2022-06-14 12:18:34: netdata ERROR : PLUGINSD[python.d] : STREAM APACHE-PROD [send]: not ready - discarding collected metrics.
2022-06-14 12:18:34: netdata INFO  : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send to YYY.YYY.YYY.YYY:19999]: initializing communication...
2022-06-14 12:18:34: netdata INFO  : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send to YYY.YYY.YYY.YYY:19999]: waiting response from remote netdata...
2022-06-14 12:18:34: netdata INFO  : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send to YYY.YYY.YYY.YYY:19999]: established communication with a parent using protocol version 4 - ready to send metrics...
2022-06-14 12:18:34: netdata ERROR : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send]: discarding 1220 bytes of metrics already in the buffer.
2022-06-14 12:18:34: netdata INFO  : PLUGINSD[ebpf] : STREAM APACHE-PROD [send]: sending metrics...
2022-06-14 12:18:35: netdata ERROR : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send to YYY.YYY.YYY.YYY:19999]: error during read (-1). Restarting connection (errno 104, Connection reset by peer)
2022-06-14 12:18:35: netdata ERROR : PLUGIN[diskspace] : STREAM APACHE-PROD [send]: not ready - discarding collected metrics.
2022-06-14 12:18:35: netdata ERROR : PLUGIN[proc] : STREAM APACHE-PROD [send]: not ready - discarding collected metrics. (errno 2, No such file or directory)
2022-06-14 12:18:35: netdata INFO  : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send to YYY.YYY.YYY.YYY:19999]: connecting...
2022-06-14 12:18:35: netdata INFO  : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send to YYY.YYY.YYY.YYY:19999]: initializing communication...
2022-06-14 12:18:35: netdata INFO  : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send to YYY.YYY.YYY.YYY:19999]: waiting response from remote netdata...
2022-06-14 12:18:35: netdata INFO  : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send to YYY.YYY.YYY.YYY:19999]: established communication with a parent using protocol version 4 - ready to send metrics...
2022-06-14 12:18:35: netdata ERROR : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send]: discarding 1416 bytes of metrics already in the buffer.
2022-06-14 12:18:35: netdata INFO  : PLUGIN[cgroups] : STREAM APACHE-PROD [send]: sending metrics...
2022-06-14 12:18:36: netdata ERROR : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send to YYY.YYY.YYY.YYY:19999]: connection closed by far end. Restarting connection
2022-06-14 12:18:36: netdata ERROR : GLOBAL_STATS : STREAM APACHE-PROD [send]: not ready - discarding collected metrics.
2022-06-14 12:18:36: netdata INFO  : STREAM_SENDER[APACHE-PROD] : STREAM APACHE-PROD [send to YYY.YYY.YYY.YYY:19999]: connecting...

Netdata version on the Netdata master: netdata v1.33.0-34-g3a8d03c3d
Netdata version on the other server: netdata v1.33.1-129-nightly

I tried to delete all the Netdata database on the Master, but it didn’t solve the issue.

Does anyone has an idea how to fix this ?

Thanks in advance.

2

Hello @DeWaRs1206,

  1. Rule of thumb keep the parent-children in almost the same version but the parent should always be on the most recent version above all the Agents.
  2. I can’t be sure that you bump into a bug or a limitation. Why don’t you try update the parent to the latest stable? (v1.35.1).

Tasos

3

Hello @Tasos_Katsoulas ,

Thanks for the answer. I have upgrade both serveur to netdata v1.35.1, let’s see if this fix the issue.

Regards

Manu

1 Like
4

After 2 weeks, no disconnection anymore. I feel a bit stupid :frowning: Thanks for the answer !

1 Like