I have a Netdata (v1.39.1) parent host collecting data from ~300 hosts (mostly v1.37.1).
Currently data collection works well but via HTTP.
Due to current security trends I would like to move to SSL.
As it is not that easy to reconfigure all 300 hosts in one moment I would like to apply “^SSL=optional”
to have both HTTP and HTTPS streaming. As soon as all hosts are SSLed I switch to “^SSL=force”.
Here is HTTP config. All works well here.
[web]
default port = 19999
bind to = *:http=dashboard|management|netdata.conf netdata.mydomain.com:19999=streaming|registry
web files owner = root
web files group = netdata
enable web responses gzip compression = no
# ssl key = /etc/netdata/ssl/mydomain.com.key
# ssl certificate = /etc/netdata/ssl/mydomain.com.crt
and here is new one. The only change is “^SSL=optional” and certs added.
[web]
default port = 19999
bind to = *:http=dashboard|management|netdata.conf netdata.mydomain.com:19999=streaming|registry^SSL=optional
web files owner = root
web files group = netdata
enable web responses gzip compression = no
ssl key = /etc/netdata/ssl/mydomain.com.key
ssl certificate = /etc/netdata/ssl/mydomain.com.crt
As soon as I activate new config all streams to parent host stop working.
Here is what child (UTC time zone) says about parent:
2023-05-30 09:29:25: netdata ERROR : STREAM_SENDER[systemAAc01.mydomain.com] : Clearing stream_collected_metrics flag in charts of host systemAAc01.mydomain.com
2023-05-30 09:29:25: netdata INFO : STREAM_SENDER[systemAAc01.mydomain.com] : STREAM systemAAc01.mydomain.com: attempting to connect to 'netdata.mydomain.com' (default port: 19999)...
2023-05-30 09:29:25: netdata INFO : STREAM_SENDER[systemAAc01.mydomain.com] : STREAM systemAAc01.mydomain.com [send to netdata.mydomain.com]: initializing communication...
2023-05-30 09:29:25: netdata INFO : STREAM_SENDER[systemAAc01.mydomain.com] : STREAM systemAAc01.mydomain.com [send to netdata.mydomain.com]: waiting response from remote netdata...
2023-05-30 09:29:26: netdata ERROR : STREAM_SENDER[systemAAc01.mydomain.com] : Clearing stream_collected_metrics flag in charts of host systemAAc01.mydomain.com
2023-05-30 09:29:26: netdata ERROR : STREAM_SENDER[systemAAc01.mydomain.com] : STREAM systemAAc01.mydomain.com [send to netdata.mydomain.com]: remote netdata does not respond.
2023-05-30 09:29:31: netdata ERROR : STREAM_SENDER[systemAAc01.mydomain.com] : Clearing stream_collected_metrics flag in charts of host systemAAc01.mydomain.com
2023-05-30 09:29:31: netdata INFO : STREAM_SENDER[systemAAc01.mydomain.com] : STREAM systemAAc01.mydomain.com: skipping destination 'netdata.mydomain.com' (default port: 19999) due to last error (code: -6, timeout while expecting first response), will retry it in 25 seconds
And here is parent log (CET time zone):
2023-05-30 11:29:26: netdata INFO : RCVR[systemAA : thread created with task id 83125
2023-05-30 11:29:26: netdata INFO : RCVR[systemAA : set name of thread 83125 to RCVR[systemAA
2023-05-30 11:29:26: netdata INFO : RCVR[systemAA : STREAM systemAAc01.mydomain.com [11.11.11.20]:43476: receive thread created (task id 83125)
2023-05-30 11:29:26: netdata INFO : RCVR[systemAA : Host systemAAc01.mydomain.com is not in archived mode anymore
2023-05-30 11:29:26: netdata INFO : RCVR[systemAA : STREAM systemAAc01.mydomain.com [receive from [11.11.11.20]:43476]: established link with negotiated capabilities: VCAPS HLABELS CLAIM CLABELS FUNCTIONS REPLICATION BINARY
2023-05-30 11:29:26: netdata ERROR : RCVR[systemAA : cannot write to SSL connection - connection is not ready.
2023-05-30 11:29:26: netdata INFO : RCVR[systemAA : STREAM 'systemAAc01.mydomain.com' [receive from [11.11.11.20]:43476]: cannot reply back. STATUS: CANT REPLY DROPPING CONNECTION
2023-05-30 11:29:26: netdata INFO : RCVR[systemAA : STREAM 'systemAAc01.mydomain.com' [receive from [11.11.11.20]:43476]: receive thread ended (task id 83125)
2023-05-30 11:29:26: netdata INFO : RCVR[systemAA : thread with task id 83125 finished
As you can see child and parent negotiated capabilities, thus I assume HTTP connection setup was completed successfully.
But suddenly parent decides to talk back SSL to client: “cannot write to SSL connection - connection is not ready”.
At this stage all falling apart and no streaming works.
Any hints what can be wrong with SSL configuration here?
Environment/Browser/Agent’s version etc
Parent streaming configuration:
$ cat /etc/netdata/stream.conf
[f1020749-6916-40fe-a425-ee2fc5fcaaf5]
enabled = yes
allow from = *
default postpone alarms on connect seconds = 300
enable compression = yes
$ netdata -W buildinfo
Version: netdata v1.39.1
Configure options: '--prefix=/usr' '--sbindir=/usr/bin' '--sysconfdir=/etc' '--libexecdir=/usr/lib' '--localstatedir=/var' '--with-zlib' '--with-math' '--with-user=netdata' 'CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -g -ffile-prefix-map=/build/netdata/src=/usr/src/debug/netdata -flto=auto' 'LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -flto=auto' 'CXXFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -Wp,-D_GLIBCXX_ASSERTIONS -g -ffile-prefix-map=/build/netdata/src=/usr/src/debug/netdata -flto=auto'
Install type: custom
Features:
dbengine: YES
Native HTTPS: YES
Netdata Cloud: YES
ACLK: YES
TLS Host Verification: YES
Machine Learning: YES
Stream Compression: YES
Libraries:
protobuf: YES (system)
jemalloc: NO
JSON-C: YES
libcap: YES
libcrypto: YES
libm: YES
tcalloc: NO
zlib: YES
Plugins:
apps: YES
cgroup Network Tracking: YES
CUPS: YES
EBPF: NO
IPMI: NO
NFACCT: YES
perf: YES
slabinfo: YES
Xen: NO
Xen VBD Error Tracking: NO
Exporters:
AWS Kinesis: NO
GCP PubSub: NO
MongoDB: YES
Prometheus Remote Write: YES
Debug/Developer Features:
Trace Allocations: NO
$ openssl version
OpenSSL 3.0.8 7 Feb 2023 (Library: OpenSSL 3.0.8 7 Feb 2023)
$ ldd /usr/bin/netdata | grep ssl
libssl.so.3 => /usr/lib/libssl.so.3 (0x00007f3b53d38000)
Child configuration:
$ cat /etc/netdata/netdata.conf
[global]
memory mode = none
update every = 4
[web]
mode = none
[registry]
enabled = no
registry to announce = http://netdata.mydomain.com:19999
[statsd]
enabled = no
$ cat /etc/netdata/stream.conf
[stream]
enabled = yes
destination = netdata.mydomain.com
api key = f1020749-6916-40fe-a425-ee2fc5fcaaf5
# netdata -W buildinfo
Version: netdata v1.37.1
Configure options: '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--program-prefix=' '--disable-dependency-tracking' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--localstatedir=/var' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--enable-plugin-freeipmi' '--with-bundled-protobuf' '--with-zlib' '--with-math' '--with-user=netdata' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'CFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic' 'LDFLAGS=-Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld' 'CXXFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic' 'PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig'
Install type: custom
Features:
dbengine: YES
Native HTTPS: YES
Netdata Cloud: YES
ACLK: YES
TLS Host Verification: YES
Machine Learning: YES
Stream Compression: NO
Libraries:
protobuf: YES (bundled)
jemalloc: NO
JSON-C: YES
libcap: YES
libcrypto: YES
libm: YES
tcalloc: NO
zlib: YES
Plugins:
apps: YES
cgroup Network Tracking: YES
CUPS: NO
EBPF: NO
IPMI: YES
NFACCT: NO
perf: YES
slabinfo: YES
Xen: NO
Xen VBD Error Tracking: NO
Exporters:
AWS Kinesis: NO
GCP PubSub: NO
MongoDB: NO
Prometheus Remote Write: YES
Debug/Developer Features:
Trace Allocations: NO
$ openssl version
OpenSSL 1.0.2k-fips 26 Jan 2017
$ ldd /sbin/netdata | grep ssl
libssl.so.10 => /lib64/libssl.so.10 (0x00007faa5d656000)