Netdata Proxy issue(s?) - remote server rejected this stream, the host we are trying to stream is already streamed to it

Help
,
1

On a netdata-parent-proxy host from an on-prem installation with somewhat latent connectivity, with latency ranging from 0-12seconds…

Which streaming proxy parameters should I tweak in order to have a more forgiving timeout policy?

[root@Netdata-Proxy netdata]$ /opt/netdata/bin/srv/netdata -V
netdata v1.42.4
[root@Netdata-Proxy netdata]$ /opt/netdata/bin/srv/netdata -W buildinfo
Packaging:
    Netdata Version ____________________________________________ : v1.42.4
    Installation Type __________________________________________ : manual-static
    Package Architecture _______________________________________ : x86_64
    Package Distro _____________________________________________ : unknown
    Configure Options __________________________________________ :  '--prefix=/opt/netdata/usr' '--sysconfdir=/opt/netdata/etc' '--localstatedir=/opt/netdata/var' '--libexecdir=/opt/netdata/usr/libexec' '--libdir=/opt/netdata/usr/lib' '--with-zlib' '--with-math' '--with-user=netdata' '--enable-cloud' '--without-bundled-protobuf' '--disable-dependency-tracking' '--enable-lto' 'CFLAGS=-ffunction-sections -fdata-sections -static -O2 -funroll-loops -I/openssl-static/include -I/libnetfilter-acct-static/include/libnetfilter_acct -I/usr/include/libmnl -pipe' 'LDFLAGS=-Wl,--gc-sections -Wl,-z,stack-size=8388608 -static -L/openssl-static/lib64 -L/libnetfilter-acct-static/lib -lnetfilter_acct -L/usr/lib -lmnl' 'PKG_CONFIG=pkg-config --static' 'PKG_CONFIG_PATH=/openssl-static/lib64/pkgconfig:/libnetfilter-acct-static/lib/pkgconfig:/usr/lib/pkgconfig'
Default Directories:
    User Configurations ________________________________________ : /opt/netdata/etc/netdata
    Stock Configurations _______________________________________ : /opt/netdata/usr/lib/netdata/conf.d
    Ephemeral Databases (metrics data, metadata) _______________ : /opt/netdata/var/cache/netdata
    Permanent Databases ________________________________________ : /opt/netdata/var/lib/netdata
    Plugins ____________________________________________________ : /opt/netdata/usr/libexec/netdata/plugins.d
    Static Web Files ___________________________________________ : /opt/netdata/usr/share/netdata/web
    Log Files __________________________________________________ : /opt/netdata/var/log/netdata
    Lock Files _________________________________________________ : /opt/netdata/var/lib/netdata/lock
    Home _______________________________________________________ : /opt/netdata/var/lib/netdata
Operating System:
    Kernel _____________________________________________________ : Linux
    Kernel Version _____________________________________________ : 4.18.0-348.el8.x86_64
    Operating System ___________________________________________ : Red Hat Enterprise Linux
    Operating System ID ________________________________________ : rhel
    Operating System ID Like ___________________________________ : fedora
    Operating System Version ___________________________________ : 8.5 (Ootpa)
    Operating System Version ID ________________________________ : none
    Detection __________________________________________________ : /etc/os-release
Hardware:
    CPU Cores __________________________________________________ : 1
    CPU Frequency ______________________________________________ : 2095000000
    CPU Architecture ___________________________________________ : 8146165760
    RAM Bytes __________________________________________________ : 85899345920
    Disk Capacity ______________________________________________ : x86_64
    Virtualization Technology __________________________________ : vmware
    Virtualization Detection ___________________________________ : systemd-detect-virt
Container:
    Container __________________________________________________ : none
    Container Detection ________________________________________ : systemd-detect-virt
    Container Orchestrator _____________________________________ : none
    Container Operating System _________________________________ : none
    Container Operating System ID ______________________________ : none
    Container Operating System ID Like _________________________ : none
    Container Operating System Version _________________________ : none
    Container Operating System Version ID ______________________ : none
    Container Operating System Detection _______________________ : none
Features:
    Built For __________________________________________________ : Linux
    Netdata Cloud ______________________________________________ : YES
    Health (trigger alerts and send notifications) _____________ : YES
    Streaming (stream metrics to parent Netdata servers) _______ : YES
    Replication (fill the gaps of parent Netdata servers) ______ : YES
    Streaming and Replication Compression ______________________ : YES (lz4)
    Contexts (index all active and archived metrics) ___________ : YES
    Tiering (multiple dbs with different metrics resolution) ___ : YES (5)
    Machine Learning ___________________________________________ : YES
Database Engines:
    dbengine ___________________________________________________ : YES
    alloc ______________________________________________________ : YES
    ram ________________________________________________________ : YES
    map ________________________________________________________ : YES
    save _______________________________________________________ : YES
    none _______________________________________________________ : YES
Connectivity Capabilities:
    ACLK (Agent-Cloud Link: MQTT over WebSockets over TLS) _____ : YES
    static (Netdata internal web server) _______________________ : YES
    h2o (web server) ___________________________________________ : YES
    WebRTC (experimental) ______________________________________ : NO
    Native HTTPS (TLS Support) _________________________________ : YES
    TLS Host Verification ______________________________________ : YES
Libraries:
    LZ4 (extremely fast lossless compression algorithm) ________ : YES
    zlib (lossless data-compression library) ___________________ : YES
    Judy (high-performance dynamic arrays and hashtables) ______ : YES (bundled)
    dlib (robust machine learning toolkit) _____________________ : YES (bundled)
    protobuf (platform-neutral data serialization protocol) ____ : YES (system)
    OpenSSL (cryptography) _____________________________________ : YES
    libdatachannel (stand-alone WebRTC data channels) __________ : NO
    JSON-C (lightweight JSON manipulation) _____________________ : YES
    libcap (Linux capabilities system operations) ______________ : NO
    libcrypto (cryptographic functions) ________________________ : YES
    libm (mathematical functions) ______________________________ : YES
    jemalloc ___________________________________________________ : NO
    TCMalloc ___________________________________________________ : NO
Plugins:
    apps (monitor processes) ___________________________________ : YES
    cgroups (monitor containers and VMs) _______________________ : YES
    cgroup-network (associate interfaces to CGROUPS) ___________ : YES
    proc (monitor Linux systems) _______________________________ : YES
    tc (monitor Linux network QoS) _____________________________ : YES
    diskspace (monitor Linux mount points) _____________________ : YES
    freebsd (monitor FreeBSD systems) __________________________ : NO
    macos (monitor MacOS systems) ______________________________ : NO
    statsd (collect custom application metrics) ________________ : YES
    timex (check system clock synchronization) _________________ : YES
    idlejitter (check system latency and jitter) _______________ : YES
    bash (support shell data collection jobs - charts.d) _______ : YES
    debugfs (kernel debugging metrics) _________________________ : YES
    cups (monitor printers and print jobs) _____________________ : NO
    ebpf (monitor system calls) ________________________________ : YES
    freeipmi (monitor enterprise server H/W) ___________________ : NO
    nfacct (gather netfilter accounting) _______________________ : YES
    perf (collect kernel performance events) ___________________ : YES
    slabinfo (monitor kernel object caching) ___________________ : YES
    Xen ________________________________________________________ : NO
    Xen VBD Error Tracking _____________________________________ : NO
Exporters:
    AWS Kinesis ________________________________________________ : NO
    GCP PubSub _________________________________________________ : NO
    MongoDB ____________________________________________________ : NO
    Prometheus (OpenMetrics) Exporter __________________________ : YES
    Prometheus Remote Write ____________________________________ : YES
    Graphite ___________________________________________________ : YES
    Graphite HTTP / HTTPS ______________________________________ : YES
    JSON _______________________________________________________ : YES
    JSON HTTP / HTTPS __________________________________________ : YES
    OpenTSDB ___________________________________________________ : YES
    OpenTSDB HTTP / HTTPS ______________________________________ : YES
    All Metrics API ____________________________________________ : YES
    Shell (use metrics in shell scripts) _______________________ : YES
Debug/Developer Features:
    Trace All Netdata Allocations (with charts) ________________ : NO
    Developer Mode (more runtime checks, slower) _______________ : NO
[root@Netdata-Proxy netdata]$ uname -a
Linux localhost.localdomain 4.18.0-348.el8.x86_64 #1 SMP Mon Oct 4 12:17:22 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux
[root@Netdata-Proxy netdata]$

Netdata Streaming Proxy Problem: I’m seeing several error codes:

(errno 9, Bad file descriptor)
(errno 104, Connection reset by peer)
(errno 22, Invalid argument)

… on the netdata parent-proxy-streaming-node.

2023-11-07 07:07:50: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: failed to send metrics - closing connection - we have sent 3369603 bytes on this connection. (errno 104, Connection reset by peer)
2023-11-07 07:07:50: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: error during receive (-1) - closing connection. (errno 9, Bad file descriptor)
2023-11-07 07:07:50: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: restarting connection: socket reports errors (POLLERR) - 3369603 bytes transmitted.
2023-11-07 07:07:50: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: remote server rejected this stream, the host we are trying to stream is already streamed to it - will retry in 120 secs, at 2023-11-07 07:09:50 (errno 22, Invalid argument)
2023-11-07 07:07:50: netdata ERROR : RCVR[Production : STREAM Production-child-1 [send]: not ready - collected metrics are not sent to parent.
2023-11-07 07:22:58: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: remote netdata does not respond.
2023-11-07 07:24:56: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: failed to send metrics - closing connection - we have sent 3442819 bytes on this connection. (errno 104, Connection reset by peer)
2023-11-07 07:24:56: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: error during receive (-1) - closing connection. (errno 9, Bad file descriptor)
2023-11-07 07:24:56: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: restarting connection: socket reports errors (POLLERR) - 3442819 bytes transmitted.
2023-11-07 07:24:56: netdata ERROR : RCVR[Production : STREAM Production-child-1 [send]: not ready - collected metrics are not sent to parent.
2023-11-07 07:25:56: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: remote netdata does not respond.
2023-11-07 07:27:26: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: remote netdata does not respond.
2023-11-07 07:28:57: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: remote netdata does not respond.
2023-11-07 07:30:29: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: remote netdata does not respond.
2023-11-07 07:32:00: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: remote netdata does not respond.
2023-11-07 07:44:18: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: failed to send metrics - closing connection - we have sent 6953408 bytes on this connection. (errno 104, Connection reset by peer)
2023-11-07 07:44:18: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: error during receive (-1) - closing connection. (errno 9, Bad file descriptor)
2023-11-07 07:44:18: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: restarting connection: socket reports errors (POLLERR) - 6953408 bytes transmitted.
2023-11-07 07:44:18: netdata ERROR : RCVR[Production : STREAM Production-child-1 [send]: not ready - collected metrics are not sent to parent.
2023-11-07 07:46:56: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: connection closed by far end. (errno 22, Invalid argument)
2023-11-07 07:46:56: netdata ERROR : RCVR[Production : STREAM Production-child-1 [send]: not ready - collected metrics are not sent to parent.
2023-11-07 07:48:06: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: connection closed by far end. (errno 22, Invalid argument)
2023-11-07 07:48:06: netdata ERROR : RCVR[Production : STREAM Production-child-1 [send]: not ready - collected metrics are not sent to parent.
2023-11-07 08:02:18: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: connection closed by far end. (errno 22, Invalid argument)
2023-11-07 08:02:18: netdata ERROR : RCVR[Production : STREAM Production-child-1 [send]: not ready - collected metrics are not sent to parent.
2023-11-07 08:03:39: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: connection closed by far end. (errno 22, Invalid argument)
2023-11-07 08:03:39: netdata ERROR : RCVR[Production : STREAM Production-child-1 [send]: not ready - collected metrics are not sent to parent.
2023-11-07 08:04:59: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: could not send metrics for 60 seconds - closing connection - we have sent 7194882 bytes on this connection via 318 send attempts. (errno 22, Invalid argument)
2023-11-07 08:04:59: netdata ERROR : RCVR[Production : STREAM Production-child-1 [send]: not ready - collected metrics are not sent to parent.
2023-11-07 08:06:11: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: could not send metrics for 60 seconds - closing connection - we have sent 1828627 bytes on this connection via 214 send attempts. (errno 22, Invalid argument)
2023-11-07 08:06:12: netdata ERROR : RCVR[Production : STREAM Production-child-1 [send]: not ready - collected metrics are not sent to parent.
2023-11-07 08:06:39: netdata ERROR : PREDICT : STREAM Production-child-1 [send]: not ready - collected metrics are not sent to parent.
2023-11-07 08:23:01: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: remote netdata does not respond.
2023-11-07 08:27:02: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: failed to send metrics - closing connection - we have sent 32296029 bytes on this connection. (errno 104, Connection reset by peer)
2023-11-07 08:27:02: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: error during receive (-1) - closing connection. (errno 9, Bad file descriptor)
2023-11-07 08:27:02: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: restarting connection: socket reports errors (POLLERR) - 32296029 bytes transmitted.
2023-11-07 08:27:02: netdata ERROR : RCVR[Production : STREAM Production-child-1 [send]: not ready - collected metrics are not sent to parent.
2023-11-07 08:30:02: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: connection closed by far end. (errno 22, Invalid argument)
2023-11-07 08:30:02: netdata ERROR : RCVR[Production : STREAM Production-child-1 [send]: not ready - collected metrics are not sent to parent.
2023-11-07 08:32:36: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: connection closed by far end. (errno 22, Invalid argument)
2023-11-07 08:32:36: netdata ERROR : RCVR[Production : STREAM Production-child-1 [send]: not ready - collected metrics are not sent to parent.
2023-11-07 08:37:40: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: connection closed by far end. (errno 22, Invalid argument)
2023-11-07 08:37:40: netdata ERROR : RCVR[Production : STREAM Production-child-1 [send]: not ready - collected metrics are not sent to parent.
2023-11-07 08:41:34: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: connection closed by far end. (errno 22, Invalid argument)
2023-11-07 08:41:34: netdata ERROR : RCVR[Production : STREAM Production-child-1 [send]: not ready - collected metrics are not sent to parent.
2023-11-07 08:43:34: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: could not send metrics for 60 seconds - closing connection - we have sent 4227712 bytes on this connection via 194 send attempts. (errno 22, Invalid argument)
2023-11-07 08:43:34: netdata ERROR : RCVR[Production : STREAM Production-child-1 [send]: not ready - collected metrics are not sent to parent.
2023-11-07 08:45:08: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: failed to send metrics - closing connection - we have sent 28098944 bytes on this connection. (errno 104, Connection reset by peer)
2023-11-07 08:45:08: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: error during receive (-1) - closing connection. (errno 9, Bad file descriptor)
2023-11-07 08:45:08: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: restarting connection: socket reports errors (POLLERR) - 28098944 bytes transmitted.
2023-11-07 08:45:08: netdata ERROR : RCVR[Production : STREAM Production-child-1 [send]: not ready - collected metrics are not sent to parent.
2023-11-07 08:45:29: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: remote server rejected this stream, the host we are trying to stream is already streamed to it - will retry in 120 secs, at 2023-11-07 08:47:29 (errno 22, Invalid argument)
2023-11-07 08:50:50: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: connection closed by far end. (errno 22, Invalid argument)
2023-11-07 08:50:50: netdata ERROR : RCVR[Production : STREAM Production-child-1 [send]: not ready - collected metrics are not sent to parent.
2023-11-07 08:52:41: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: could not send metrics for 60 seconds - closing connection - we have sent 7111763 bytes on this connection via 288 send attempts. (errno 22, Invalid argument)
2023-11-07 08:52:41: netdata ERROR : PREDICT : STREAM Production-child-1 [send]: not ready - collected metrics are not sent to parent.
2023-11-07 08:56:04: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: connection closed by far end. (errno 22, Invalid argument)
2023-11-07 08:56:04: netdata ERROR : RCVR[Production : STREAM Production-child-1 [send]: not ready - collected metrics are not sent to parent.
2023-11-07 08:57:25: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: failed to send metrics - closing connection - we have sent 20534912 bytes on this connection. (errno 104, Connection reset by peer)
2023-11-07 08:57:25: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: error during receive (-1) - closing connection. (errno 9, Bad file descriptor)
2023-11-07 08:57:25: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: restarting connection: socket reports errors (POLLERR) - 20534912 bytes transmitted.
2023-11-07 08:57:25: netdata ERROR : RCVR[Production : STREAM Production-child-1 [send]: not ready - collected metrics are not sent to parent.
2023-11-07 09:03:45: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: remote netdata does not respond.
2023-11-07 09:09:47: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: connection closed by far end. (errno 22, Invalid argument)
2023-11-07 09:09:47: netdata ERROR : RCVR[Production : STREAM Production-child-1 [send]: not ready - collected metrics are not sent to parent.
2023-11-07 09:16:39: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: failed to send metrics - closing connection - we have sent 160269619 bytes on this connection. (errno 104, Connection reset by peer)
2023-11-07 09:16:39: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: error during receive (-1) - closing connection. (errno 9, Bad file descriptor)
2023-11-07 09:16:39: netdata ERROR : RCVR[Production : STREAM Production-child-1 [send]: not ready - collected metrics are not sent to parent.
2023-11-07 09:16:39: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: restarting connection: socket reports errors (POLLERR) - 160269619 bytes transmitted.
2023-11-07 09:24:52: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: failed to send metrics - closing connection - we have sent 8176363 bytes on this connection. (errno 104, Connection reset by peer)
2023-11-07 09:24:52: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: error during receive (-1) - closing connection. (errno 9, Bad file descriptor)
2023-11-07 09:24:52: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: restarting connection: socket reports errors (POLLERR) - 8176363 bytes transmitted.
2023-11-07 09:24:52: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: remote server rejected this stream, the host we are trying to stream is already streamed to it - will retry in 120 secs, at 2023-11-07 09:26:52 (errno 22, Invalid argument)
2023-11-07 09:24:52: netdata ERROR : RCVR[Production : STREAM Production-child-1 [send]: not ready - collected metrics are not sent to parent.
2023-11-07 09:29:15: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: failed to send metrics - closing connection - we have sent 36846749 bytes on this connection. (errno 104, Connection reset by peer)
2023-11-07 09:29:15: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: error during receive (-1) - closing connection. (errno 9, Bad file descriptor)
2023-11-07 09:29:15: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: restarting connection: socket reports errors (POLLERR) - 36846749 bytes transmitted.
2023-11-07 09:29:15: netdata ERROR : RCVR[Production : STREAM Production-child-1 [send]: not ready - collected metrics are not sent to parent.
2023-11-07 09:29:16: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: remote server rejected this stream, the host we are trying to stream is already streamed to it - will retry in 120 secs, at 2023-11-07 09:31:16 (errno 22, Invalid argument)
2023-11-07 09:33:15: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: connection closed by far end. (errno 22, Invalid argument)
2023-11-07 09:33:15: netdata ERROR : RCVR[Production : STREAM Production-child-1 [send]: not ready - collected metrics are not sent to parent.
2023-11-07 09:34:06: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: failed to send metrics - closing connection - we have sent 16556041 bytes on this connection. (errno 104, Connection reset by peer)
2023-11-07 09:34:06: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: error during receive (-1) - closing connection. (errno 9, Bad file descriptor)
2023-11-07 09:34:06: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: restarting connection: socket reports errors (POLLERR) - 16556041 bytes transmitted.
2023-11-07 09:34:06: netdata ERROR : RCVR[Production : STREAM Production-child-1 [send]: not ready - collected metrics are not sent to parent.
2023-11-07 09:34:06: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: remote server rejected this stream, the host we are trying to stream is already streamed to it - will retry in 120 secs, at 2023-11-07 09:36:06 (errno 22, Invalid argument)
2023-11-07 09:37:28: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: failed to send metrics - closing connection - we have sent 23372290 bytes on this connection. (errno 104, Connection reset by peer)
2023-11-07 09:37:28: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: error during receive (-1) - closing connection. (errno 9, Bad file descriptor)
2023-11-07 09:37:28: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: restarting connection: socket reports errors (POLLERR) - 23372290 bytes transmitted.
2023-11-07 09:37:28: netdata ERROR : RCVR[Production : STREAM Production-child-1 [send]: not ready - collected metrics are not sent to parent.
2023-11-07 09:40:19: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: connection closed by far end. (errno 22, Invalid argument)
2023-11-07 09:40:19: netdata ERROR : RCVR[Production : STREAM Production-child-1 [send]: not ready - collected metrics are not sent to parent.
2023-11-07 09:45:11: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: connection closed by far end. (errno 22, Invalid argument)
2023-11-07 09:45:11: netdata ERROR : RCVR[Production : STREAM Production-child-1 [send]: not ready - collected metrics are not sent to parent.
2023-11-07 09:47:43: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: connection closed by far end. (errno 22, Invalid argument)
2023-11-07 09:47:43: netdata ERROR : RCVR[Production : STREAM Production-child-1 [send]: not ready - collected metrics are not sent to parent.
2023-11-07 09:48:43: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: remote netdata does not respond.
2023-11-07 10:10:37: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: connection closed by far end. (errno 22, Invalid argument)
2023-11-07 10:10:37: netdata ERROR : RCVR[Production : STREAM Production-child-1 [send]: not ready - collected metrics are not sent to parent.
2023-11-07 10:11:57: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: failed to send metrics - closing connection - we have sent 14040064 bytes on this connection. (errno 104, Connection reset by peer)
2023-11-07 10:11:57: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: error during receive (-1) - closing connection. (errno 9, Bad file descriptor)
2023-11-07 10:11:57: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: restarting connection: socket reports errors (POLLERR) - 14040064 bytes transmitted.
2023-11-07 10:11:57: netdata ERROR : RCVR[Production : STREAM Production-child-1 [send]: not ready - collected metrics are not sent to parent.
2023-11-07 10:16:48: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: connection closed by far end. (errno 22, Invalid argument)
2023-11-07 10:16:48: netdata ERROR : RCVR[Production : STREAM Production-child-1 [send]: not ready - collected metrics are not sent to parent.
2023-11-07 10:23:21: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: connection closed by far end. (errno 22, Invalid argument)
2023-11-07 10:23:21: netdata ERROR : RCVR[Production : STREAM Production-child-1 [send]: not ready - collected metrics are not sent to parent.
2023-11-07 10:25:42: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: connection closed by far end. (errno 22, Invalid argument)
2023-11-07 10:25:42: netdata ERROR : RCVR[Production : STREAM Production-child-1 [send]: not ready - collected metrics are not sent to parent.
2023-11-07 10:26:43: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: remote netdata does not respond.
2023-11-07 10:28:16: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: remote netdata does not respond.
2023-11-07 10:29:46: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: remote netdata does not respond.
2023-11-07 10:31:16: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: remote netdata does not respond.
2023-11-07 10:37:02: netdata ERROR : SNDR[Production : STREAM Production-child-1 [send to tcp:netdata.example.com:19999]: connection closed by far end. (errno 22, Invalid argument)

What I’ve tried so far unsuccessfully

  • double/triple/quadruple buffer sizes
  • enabled/disabled compression
  • disabled SSL/TLS1.3
2

Hi @mitsuto

Sorry for the late reply.

Indeed, streaming could suffer on a slow/big latency link. I did some tests on a simulated ~10 second latency, and of course real time streaming of metrics is not possible. However, it still depends. On my case some metrics do come in, some don’t. It would also depend on the speed of the link.

I didn’t get the the host we are trying to stream is already streamed to it. Likely the cause here is a dropped connection where the parent is not aware it being off yet. There is a timeout there, so at some point the child will reconnect.

I’m sorry I can’t offer a solution at this point, I see you mostly tried everything. Maybe suggest to try zstd compression which was a recent addition, to check if it makes much difference.

We will update here if we have something more…

Thanks.

3

Hi Manolis, thanks for your reply albeit late.

What’s zstd compression? I already have enable compression = yes in stream.conf. Are there undocumented features I could enable? Or would I have to install from the nightly release channel?

4

Yes, zstd compression has been in the latest nightly releases.

If you use a nightly, then using edit-config to start a new stream.conf, it should have: compression algorithms order = zstd lz4 brotli gzip option.

5

ok, I’ll give it a try.