Mixed plain and SSLed data streaming issue

Help
,
1

I have a Netdata (v1.39.1) parent host collecting data from ~300 hosts (mostly v1.37.1).
Currently data collection works well but via HTTP.
Due to current security trends I would like to move to SSL.

As it is not that easy to reconfigure all 300 hosts in one moment I would like to apply “^SSL=optional”
to have both HTTP and HTTPS streaming. As soon as all hosts are SSLed I switch to “^SSL=force”.

Here is HTTP config. All works well here.

[web]
    default port = 19999
    bind to = *:http=dashboard|management|netdata.conf netdata.mydomain.com:19999=streaming|registry
    web files owner = root
    web files group = netdata
    enable web responses gzip compression = no
#    ssl key = /etc/netdata/ssl/mydomain.com.key
#    ssl certificate = /etc/netdata/ssl/mydomain.com.crt

and here is new one. The only change is “^SSL=optional” and certs added.

[web]
    default port = 19999
    bind to = *:http=dashboard|management|netdata.conf netdata.mydomain.com:19999=streaming|registry^SSL=optional
    web files owner = root
    web files group = netdata
    enable web responses gzip compression = no
    ssl key = /etc/netdata/ssl/mydomain.com.key
    ssl certificate = /etc/netdata/ssl/mydomain.com.crt

As soon as I activate new config all streams to parent host stop working.
Here is what child (UTC time zone) says about parent:

2023-05-30 09:29:25: netdata ERROR : STREAM_SENDER[systemAAc01.mydomain.com] : Clearing stream_collected_metrics flag in charts of host systemAAc01.mydomain.com
2023-05-30 09:29:25: netdata INFO  : STREAM_SENDER[systemAAc01.mydomain.com] : STREAM systemAAc01.mydomain.com: attempting to connect to 'netdata.mydomain.com' (default port: 19999)...
2023-05-30 09:29:25: netdata INFO  : STREAM_SENDER[systemAAc01.mydomain.com] : STREAM systemAAc01.mydomain.com [send to netdata.mydomain.com]: initializing communication...
2023-05-30 09:29:25: netdata INFO  : STREAM_SENDER[systemAAc01.mydomain.com] : STREAM systemAAc01.mydomain.com [send to netdata.mydomain.com]: waiting response from remote netdata...
2023-05-30 09:29:26: netdata ERROR : STREAM_SENDER[systemAAc01.mydomain.com] : Clearing stream_collected_metrics flag in charts of host systemAAc01.mydomain.com
2023-05-30 09:29:26: netdata ERROR : STREAM_SENDER[systemAAc01.mydomain.com] : STREAM systemAAc01.mydomain.com [send to netdata.mydomain.com]: remote netdata does not respond.
2023-05-30 09:29:31: netdata ERROR : STREAM_SENDER[systemAAc01.mydomain.com] : Clearing stream_collected_metrics flag in charts of host systemAAc01.mydomain.com
2023-05-30 09:29:31: netdata INFO  : STREAM_SENDER[systemAAc01.mydomain.com] : STREAM systemAAc01.mydomain.com: skipping destination 'netdata.mydomain.com' (default port: 19999) due to last error (code: -6, timeout while expecting first response), will retry it in 25 seconds

And here is parent log (CET time zone):

2023-05-30 11:29:26: netdata INFO  : RCVR[systemAA : thread created with task id 83125
2023-05-30 11:29:26: netdata INFO  : RCVR[systemAA : set name of thread 83125 to RCVR[systemAA
2023-05-30 11:29:26: netdata INFO  : RCVR[systemAA : STREAM systemAAc01.mydomain.com [11.11.11.20]:43476: receive thread created (task id 83125)
2023-05-30 11:29:26: netdata INFO  : RCVR[systemAA : Host systemAAc01.mydomain.com is not in archived mode anymore
2023-05-30 11:29:26: netdata INFO  : RCVR[systemAA : STREAM systemAAc01.mydomain.com [receive from [11.11.11.20]:43476]: established link with negotiated capabilities: VCAPS HLABELS CLAIM CLABELS FUNCTIONS REPLICATION BINARY
2023-05-30 11:29:26: netdata ERROR : RCVR[systemAA : cannot write to SSL connection - connection is not ready.
2023-05-30 11:29:26: netdata INFO  : RCVR[systemAA : STREAM 'systemAAc01.mydomain.com' [receive from [11.11.11.20]:43476]: cannot reply back. STATUS: CANT REPLY DROPPING CONNECTION
2023-05-30 11:29:26: netdata INFO  : RCVR[systemAA : STREAM 'systemAAc01.mydomain.com' [receive from [11.11.11.20]:43476]: receive thread ended (task id 83125)
2023-05-30 11:29:26: netdata INFO  : RCVR[systemAA : thread with task id 83125 finished

As you can see child and parent negotiated capabilities, thus I assume HTTP connection setup was completed successfully.
But suddenly parent decides to talk back SSL to client: “cannot write to SSL connection - connection is not ready”.
At this stage all falling apart and no streaming works.

Any hints what can be wrong with SSL configuration here?

Environment/Browser/Agent’s version etc

Parent streaming configuration:

$ cat /etc/netdata/stream.conf
[f1020749-6916-40fe-a425-ee2fc5fcaaf5]
    enabled = yes
    allow from = *
    default postpone alarms on connect seconds = 300
    enable compression = yes

$ netdata -W buildinfo
Version: netdata v1.39.1
Configure options:  '--prefix=/usr' '--sbindir=/usr/bin' '--sysconfdir=/etc' '--libexecdir=/usr/lib' '--localstatedir=/var' '--with-zlib' '--with-math' '--with-user=netdata' 'CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions         -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security         -fstack-clash-protection -fcf-protection -g -ffile-prefix-map=/build/netdata/src=/usr/src/debug/netdata -flto=auto' 'LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -flto=auto' 'CXXFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions         -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security         -fstack-clash-protection -fcf-protection -Wp,-D_GLIBCXX_ASSERTIONS -g -ffile-prefix-map=/build/netdata/src=/usr/src/debug/netdata -flto=auto'
Install type: custom
Features:
    dbengine:                   YES
    Native HTTPS:               YES
    Netdata Cloud:              YES
    ACLK:                       YES
    TLS Host Verification:      YES
    Machine Learning:           YES
    Stream Compression:         YES
Libraries:
    protobuf:                YES (system)
    jemalloc:                NO
    JSON-C:                  YES
    libcap:                  YES
    libcrypto:               YES
    libm:                    YES
    tcalloc:                 NO
    zlib:                    YES
Plugins:
    apps:                    YES
    cgroup Network Tracking: YES
    CUPS:                    YES
    EBPF:                    NO
    IPMI:                    NO
    NFACCT:                  YES
    perf:                    YES
    slabinfo:                YES
    Xen:                     NO
    Xen VBD Error Tracking:  NO
Exporters:
    AWS Kinesis:             NO
    GCP PubSub:              NO
    MongoDB:                 YES
    Prometheus Remote Write: YES
Debug/Developer Features:
    Trace Allocations:       NO

$ openssl version
OpenSSL 3.0.8 7 Feb 2023 (Library: OpenSSL 3.0.8 7 Feb 2023)

$ ldd /usr/bin/netdata | grep ssl
        libssl.so.3 => /usr/lib/libssl.so.3 (0x00007f3b53d38000)

Child configuration:

$ cat /etc/netdata/netdata.conf
[global]
        memory mode = none
        update every = 4

[web]
        mode = none

[registry]
        enabled = no
        registry to announce = http://netdata.mydomain.com:19999

[statsd]
        enabled = no

$ cat /etc/netdata/stream.conf
[stream]
    enabled = yes
    destination = netdata.mydomain.com
    api key = f1020749-6916-40fe-a425-ee2fc5fcaaf5

# netdata -W buildinfo
Version: netdata v1.37.1
Configure options:  '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--program-prefix=' '--disable-dependency-tracking' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--localstatedir=/var' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--enable-plugin-freeipmi' '--with-bundled-protobuf' '--with-zlib' '--with-math' '--with-user=netdata' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'CFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1  -m64 -mtune=generic' 'LDFLAGS=-Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld' 'CXXFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1  -m64 -mtune=generic' 'PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig'
Install type: custom
Features:
    dbengine:                   YES
    Native HTTPS:               YES
    Netdata Cloud:              YES
    ACLK:                       YES
    TLS Host Verification:      YES
    Machine Learning:           YES
    Stream Compression:         NO
Libraries:
    protobuf:                YES (bundled)
    jemalloc:                NO
    JSON-C:                  YES
    libcap:                  YES
    libcrypto:               YES
    libm:                    YES
    tcalloc:                 NO
    zlib:                    YES
Plugins:
    apps:                    YES
    cgroup Network Tracking: YES
    CUPS:                    NO
    EBPF:                    NO
    IPMI:                    YES
    NFACCT:                  NO
    perf:                    YES
    slabinfo:                YES
    Xen:                     NO
    Xen VBD Error Tracking:  NO
Exporters:
    AWS Kinesis:             NO
    GCP PubSub:              NO
    MongoDB:                 NO
    Prometheus Remote Write: YES
Debug/Developer Features:
    Trace Allocations:       NO

$ openssl version
OpenSSL 1.0.2k-fips  26 Jan 2017

$ ldd /sbin/netdata | grep ssl
        libssl.so.10 => /lib64/libssl.so.10 (0x00007faa5d656000)
2

Hi @lex sorry for the late reply. Just to note that we are looking into it and will come back with more info. Thank you!

3

Parent update to netdata v1.40.0 greatly improved situation.
Streaming with SSL=optional works well with SSLed and plain children.
No update on children was needed, still have most on 1.37.1.