Netdata agent is unable to connect with netdata cloud

Help
1

Suggested template:

Problem/Question

Netdata agent is unable to connect with netdata cloud

Relevant docs you followed/actions you took to solve the issue

I can’t find the link now.

Environment/Browser/Agent’s version etc

Edge latest

What I expected to happen

It is supposed to connect.

I’m having the same issue on brand new ubuntu jammy installation and brand new netdata installation.

Here is the installed ca-certicates package

root@server:/opt# apt-cache policy ca-certificates
ca-certificates:
  Installed: 20211016
  Candidate: 20211016
  Version table:
 *** 20211016 500
        500 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages
        100 /var/lib/dpkg/status

aclk status

root@server:/opt# netdatacli aclk-state
ACLK Available: Yes
ACLK Version: 2
Protocols Supported: Protobuf
Protocol Used: Protobuf
MQTT Version: 5
Claimed: Yes
Claimed Id: xxxxxxxxxx-xxx-xxxx-xxxx-xxxxxxx
Cloud URL: https://app.netdata.cloud
Online: No
Reconnect count: 0
Banned By Cloud: No
Next Connection Attempt At: 2022-12-03 03:25:03
Last Backoff: 20.029

2022-12-03 03:24:24: netdata INFO : ACLK_Main : Attempting connection now
2022-12-03 03:24:24: netdata ERROR : ACLK_Main : Cert Chain verify error:num=20:unable to get local issuer certificate:depth=2:/C=US/O=Internet Security Research Group/CN=ISRG Root X1 (errno 2, No such file or directory)
2022-12-03 03:24:24: netdata ERROR : ACLK_Main : SSL_write Err: SSL_ERROR_SSL
2022-12-03 03:24:24: netdata ERROR : ACLK_Main : Couldn’t write HTTP request header into SSL connection
2022-12-03 03:24:24: netdata ERROR : ACLK_Main : Couldn’t process request
2022-12-03 03:24:24: netdata ERROR : ACLK_Main : Error trying to contact env endpoint
2022-12-03 03:24:24: netdata ERROR : ACLK_Main : Failed to Get ACLK environment
2022-12-03 03:24:24: netdata INFO : ACLK_Main : Wait before attempting to reconnect in 2.071 seconds

2022-12-03 03:24:26: netdata INFO : ACLK_Main : Attempting connection now
2022-12-03 03:24:26: netdata ERROR : ACLK_Main : Cert Chain verify error:num=20:unable to get local issuer certificate:depth=2:/C=US/O=Internet Security Research Group/CN=ISRG Root X1 (errno 2, No such file or directory)
2022-12-03 03:24:26: netdata ERROR : ACLK_Main : SSL_write Err: SSL_ERROR_SSL
2022-12-03 03:24:26: netdata ERROR : ACLK_Main : Couldn’t write HTTP request header into SSL connection
2022-12-03 03:24:26: netdata ERROR : ACLK_Main : Couldn’t process request
2022-12-03 03:24:26: netdata ERROR : ACLK_Main : Error trying to contact env endpoint
2022-12-03 03:24:26: netdata ERROR : ACLK_Main : Failed to Get ACLK environment
2022-12-03 03:24:26: netdata INFO : ACLK_Main : Wait before attempting to reconnect in 5.409 seconds

2022-12-03 03:24:32: netdata INFO : ACLK_Main : Attempting connection now
2022-12-03 03:24:32: netdata ERROR : ACLK_Main : Cert Chain verify error:num=20:unable to get local issuer certificate:depth=2:/C=US/O=Internet Security Research Group/CN=ISRG Root X1 (errno 2, No such file or directory)
2022-12-03 03:24:32: netdata ERROR : ACLK_Main : SSL_write Err: SSL_ERROR_SSL
2022-12-03 03:24:32: netdata ERROR : ACLK_Main : Couldn’t write HTTP request header into SSL connection
2022-12-03 03:24:32: netdata ERROR : ACLK_Main : Couldn’t process request
2022-12-03 03:24:32: netdata ERROR : ACLK_Main : Error trying to contact env endpoint
2022-12-03 03:24:32: netdata ERROR : ACLK_Main : Failed to Get ACLK environment
2022-12-03 03:24:32: netdata INFO : ACLK_Main : Wait before attempting to reconnect in 11.109 seconds

2022-12-03 03:24:43: netdata INFO : ACLK_Main : Attempting connection now
2022-12-03 03:24:43: netdata ERROR : ACLK_Main : Cert Chain verify error:num=20:unable to get local issuer certificate:depth=2:/C=US/O=Internet Security Research Group/CN=ISRG Root X1 (errno 2, No such file or directory)
2022-12-03 03:24:43: netdata ERROR : ACLK_Main : SSL_write Err: SSL_ERROR_SSL
2022-12-03 03:24:43: netdata ERROR : ACLK_Main : Couldn’t write HTTP request header into SSL connection
2022-12-03 03:24:43: netdata ERROR : ACLK_Main : Couldn’t process request
2022-12-03 03:24:43: netdata ERROR : ACLK_Main : Error trying to contact env endpoint
2022-12-03 03:24:43: netdata ERROR : ACLK_Main : Failed to Get ACLK environment
2022-12-03 03:24:43: netdata INFO : ACLK_Main : Wait before attempting to reconnect in 20.029 seconds

2022-12-03 03:24:45: netdata ERROR : HEALTH[GSB10G-20] : Failed to prepare statement when trying to filter alert events.
2022-12-03 03:24:45: netdata ERROR : HEALTH[GSB10G-20] : Failed to prepare statement to store alert event
2022-12-03 03:24:58: netdata INFO : MAIN : Command Clients = 1

2022-12-03 03:24:58: netdata INFO : MAIN : EOF found in command pipe.
2022-12-03 03:24:58: netdata INFO : MAIN : COMMAND: Reopening aclk/cloud state.
2022-12-03 03:24:58: netdata INFO : MAIN : COMMAND: Sending reply: “X0”
2022-12-03 03:24:58: netdata INFO : MAIN : Command Clients = 0

2022-12-03 03:25:03: netdata INFO : ACLK_Main : Attempting connection now
2022-12-03 03:25:03: netdata ERROR : ACLK_Main : Cert Chain verify error:num=20:unable to get local issuer certificate:depth=2:/C=US/O=Internet Security Research Group/CN=ISRG Root X1 (errno 2, No such file or directory)
2022-12-03 03:25:03: netdata ERROR : ACLK_Main : SSL_write Err: SSL_ERROR_SSL
2022-12-03 03:25:03: netdata ERROR : ACLK_Main : Couldn’t write HTTP request header into SSL connection
2022-12-03 03:25:03: netdata ERROR : ACLK_Main : Couldn’t process request
2022-12-03 03:25:03: netdata ERROR : ACLK_Main : Error trying to contact env endpoint
2022-12-03 03:25:03: netdata ERROR : ACLK_Main : Failed to Get ACLK environment
2022-12-03 03:25:03: netdata INFO : ACLK_Main : Wait before attempting to reconnect in 39.480 seconds

Please any help here? I tried on some other machines as well and all same issue.

2

Hello,
I was trying to replicate your issue, but it only makes me happen when I removed the Let’s Encrypt CA, so please ensure that you have that certificate.
My replication path:

# lsb_release -av
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.1 LTS
Release:	22.04
Codename:	jammy
# apt-cache policy ca-certificates
ca-certificates:
  Installed: 20211016
  Candidate: 20211016
  Version table:
 *** 20211016 500
        500 http://ports.ubuntu.com/ubuntu-ports jammy/main arm64 Packages
        100 /var/lib/dpkg/status
# curl https://app.netdata.cloud -I
HTTP/2 200
accept-ranges: bytes
cache-control: no-cache
content-length: 4779
content-type: text/html
date: Mon, 05 Dec 2022 09:41:17 GMT
etag: "63849012-12ab"
expires: Thu, 01 Jan 1970 00:00:01 GMT
last-modified: Mon, 28 Nov 2022 10:40:18 GMT
server: nginx/1.17.5
vary: Accept-Encoding
x-frame-options: SAMEORIGIN
# rm /usr/share/ca-certificates/mozilla/ISRG_Root_X1.crt
# update-ca-certificates
Updating certificates in /etc/ssl/certs...
W: /usr/share/ca-certificates/mozilla/ISRG_Root_X1.crt not found, but listed in /etc/ca-certificates.conf.
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
root@024ca515d0cb:/etc/ssl/certs# curl https://app.netdata.cloud -I
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
# tail -f /var/log/netdata/error.log
2022-12-05 09:41:46: netdata ERROR : ACLK_Main : Cert Chain verify error:num=20:unable to get local issuer certificate:depth=2:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
2022-12-05 09:41:46: netdata ERROR : ACLK_Main : SSL_write Err: SSL_ERROR_SSL
2022-12-05 09:41:46: netdata ERROR : ACLK_Main : Couldn't write HTTP request header into SSL connection
2022-12-05 09:41:46: netdata ERROR : ACLK_Main : Couldn't process request
2022-12-05 09:41:46: netdata ERROR : ACLK_Main : Error trying to contact env endpoint
2022-12-05 09:41:46: netdata ERROR : ACLK_Main : Failed to Get ACLK environment
2022-12-05 09:41:46: netdata INFO  : ACLK_Main : Wait before attempting to reconnect in 1.477 seconds
3

I already have that certificate but it seems netdata is not using that somehow…Check my output here…What am I missing here?

root@test:~#  lsb_release -av
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.1 LTS
Release:        22.04
Codename:       jammy
root@test:~# apt-cache policy ca-certificates
ca-certificates:
  Installed: 20211016ubuntu0.22.04.1
  Candidate: 20211016ubuntu0.22.04.1
  Version table:
 *** 20211016ubuntu0.22.04.1 500
        500 http://archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages
        500 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     20211016 500
        500 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages
root@test:~#  curl https://app.netdata.cloud -I
HTTP/2 200 
accept-ranges: bytes
cache-control: no-cache
content-length: 4967
content-type: text/html
date: Sat, 10 Dec 2022 17:05:15 GMT
etag: "6390b071-1367"
expires: Thu, 01 Jan 1970 00:00:01 GMT
last-modified: Wed, 07 Dec 2022 15:25:37 GMT
server: nginx/1.17.5
vary: Accept-Encoding
x-frame-options: SAMEORIGIN

root@test:~# update-ca-certificates
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...

updates of cacerts keystore disabled.
done.
root@test:~# find /etc -name ISRG
root@test:~# find /etc -name ISRG*
/etc/ssl/certs/ISRG_Root_X1.pem
root@test:~# file /etc/ssl/certs/ISRG_Root_X1.pem
/etc/ssl/certs/ISRG_Root_X1.pem: symbolic link to /usr/share/ca-certificates/mozilla/ISRG_Root_X1.crt
root@test:~# file /usr/share/ca-certificates/mozilla/ISRG_Root_X1.crt
/usr/share/ca-certificates/mozilla/ISRG_Root_X1.crt: PEM certificate
4

What is your install method/type?
Can you provide netdata -W buildinfo ?

Maybe those certificates are not loaded/used despite being installed?

We use SSL_CTX_set_default_verify_paths (docu: /docs/man3.0/man3/SSL_CTX_set_default_verify_paths.html). Which uses default OpenSSL behavior to find out the certificate list to use. Seems like maybe that fails for some reason (which is also reason why I ask about install method etc. above).

5

Hello,

Maybe those certificates are not loaded/used despite being installed?

If that’s the case How can I fix it on my own?

I installed it using netdata official install script. and here is the output.

root@test:~# netdata -W buildinfo
Version: netdata v1.37.1
Configure options: ‘–build=x86_64-linux-gnu’ ‘–includedir=${prefix}/include’
‘–mandir=${prefix}/share/man’ ‘–infodir=${prefix}/share/info’ ‘–disable-option-checking’ ‘–disable-silent-rules’ ‘–libdir=${prefix}/lib/x86_64-linux-gnu’ ‘–libexecdir=${prefix}/lib/x86_64-linux-gnu’ ‘–disable-maintainer-mode’ ‘–prefix=/usr’ ‘–sysconfdir=/etc’ ‘–localstatedir=/var’ ‘–libdir=/usr/lib’ ‘–libexecdir=/usr/libexec’ ‘–with-user=netdata’ ‘–with-math’ ‘–with-zlib’ ‘–with-webdir=/var/lib/netdata/www’ ‘–disable-dependency-tracking’ ‘build_alias=x86_64-linux-gnu’ ‘CFLAGS=-g -O2 -ffile-prefix-map=/usr/src/netdata=. -fstack-protector-strong -Wformat -Werror=format-security’ ‘LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro’ ‘CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2’ ‘CXXFLAGS=-g -O2 -ffile-prefix-map=/usr/src/netdata=. -fstack-protector-strong -Wformat -Werror=format-security’
Install type: binpkg-deb
Binary architecture: x86_64
Packaging distro:
Features:
dbengine: YES
Native HTTPS: YES
Netdata Cloud: YES
ACLK: YES
TLS Host Verification: YES
Machine Learning: YES
Stream Compression: YES
Libraries:
protobuf: YES (system)
jemalloc: NO
JSON-C: YES
libcap: NO
libcrypto: YES
libm: YES
tcalloc: NO
zlib: YES
Plugins:
apps: YES
cgroup Network Tracking: YES
CUPS: YES
EBPF: YES
IPMI: YES
NFACCT: YES
perf: YES
slabinfo: YES
Xen: NO
Xen VBD Error Tracking: NO
Exporters:
AWS Kinesis: NO
GCP PubSub: NO
MongoDB: NO
Prometheus Remote Write: YES
Debug/Developer Features:
Trace Allocations: NO

6

Seems to work on clean ubuntu 22.04.1 LTS install with same certs version.

/usr/sbin/netdata -W buildinfo
Version: netdata v1.37.1
Configure options:  '--build=x86_64-linux-gnu' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--disable-option-checking' '--disable-silent-rules' '--libdir=${prefix}/lib/x86_64-linux-gnu' '--libexecdir=${prefix}/lib/x86_64-linux-gnu' '--disable-maintainer-mode' '--prefix=/usr' '--sysconfdir=/etc' '--localstatedir=/var' '--libdir=/usr/lib' '--libexecdir=/usr/libexec' '--with-user=netdata' '--with-math' '--with-zlib' '--with-webdir=/var/lib/netdata/www' '--disable-dependency-tracking' 'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -ffile-prefix-map=/usr/src/netdata=. -fstack-protector-strong -Wformat -Werror=format-security' 'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -ffile-prefix-map=/usr/src/netdata=. -fstack-protector-strong -Wformat -Werror=format-security'
Install type: binpkg-deb
    Binary architecture: x86_64
    Packaging distro:  
Features:
    dbengine:                   YES
    Native HTTPS:               YES
    Netdata Cloud:              YES 
    ACLK:                       YES
    TLS Host Verification:      YES
    Machine Learning:           YES
    Stream Compression:         YES
Libraries:
    protobuf:                YES (system)
    jemalloc:                NO
    JSON-C:                  YES
    libcap:                  NO
    libcrypto:               YES
    libm:                    YES
    tcalloc:                 NO
    zlib:                    YES
Plugins:
    apps:                    YES
    cgroup Network Tracking: YES
    CUPS:                    YES
    EBPF:                    YES
    IPMI:                    YES
    NFACCT:                  YES
    perf:                    YES
    slabinfo:                YES
    Xen:                     NO
    Xen VBD Error Tracking:  NO
Exporters:
    AWS Kinesis:             NO
    GCP PubSub:              NO
    MongoDB:                 NO
    Prometheus Remote Write: YES
Debug/Developer Features:
    Trace Allocations:       NO

ca cert

apt-cache policy ca-certificates
ca-certificates:
  Installed: 20211016ubuntu0.22.04.1
  Candidate: 20211016ubuntu0.22.04.1
  Version table:
 *** 20211016ubuntu0.22.04.1 500
        500 http://th.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
        500 http://th.archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages
        100 /var/lib/dpkg/status
     20211016 500
        500 http://th.archive.ubuntu.com/ubuntu jammy/main amd64 Packages
7

are you able to also curl/wget https://api.netdata.cloud/api/v1/env ?
anything in the system setting env vars SSL_CERT_DIR or SSL_CERT_FILE for netdata?

8

Yes, I am able to curl/wget to that address without any SSL error. Also, netdata is started by systemd and I can’t see any environmental variable SSL_CERT_DIRor SSL_CERT_FILEfor netdata. You can check the output of systemctl cat netdata below.

root@Test:~# curl -I https://api.netdata.cloud/api/v1/env
HTTP/2 404
accept-ranges: bytes
cache-control: no-cache
content-length: 397
content-type: text/html
date: Wed, 14 Dec 2022 11:35:40 GMT
etag: “639992ba-18d”
expires: Thu, 01 Jan 1970 00:00:01 GMT
last-modified: Wed, 14 Dec 2022 09:09:14 GMT
server: nginx/1.17.5
vary: Accept-Encoding
x-frame-options: SAMEORIGIN

root@Test:~# wget -S -O - https://api.netdata.cloud/api/v1/env
–2022-12-14 17:05:48-- https://api.netdata.cloud/api/v1/env
Resolving api.netdata.cloud (api.netdata.cloud)… 44.196.50.41, 54.198.178.11, 44.207.131.212
Connecting to api.netdata.cloud (api.netdata.cloud)|44.196.50.41|:443… connected.
HTTP request sent, awaiting response…
HTTP/1.1 400 Bad Request
content-length: 145
content-type: application/json; charset=utf-8
date: Wed, 14 Dec 2022 11:35:48 GMT
vary: Accept-Encoding
2022-12-14 17:05:48 ERROR 400: Bad Request.

root@Test:~# wget -O - https://api.netdata.cloud/api/v1/env
–2022-12-14 17:05:56-- https://api.netdata.cloud/api/v1/env
Resolving api.netdata.cloud (api.netdata.cloud)… 44.196.50.41, 44.207.131.212, 54.198.178.11
Connecting to api.netdata.cloud (api.netdata.cloud)|44.196.50.41|:443… connected.
HTTP request sent, awaiting response… 400 Bad Request
2022-12-14 17:05:57 ERROR 400: Bad Request.

root@Test:~# systemctl cat netdata

/lib/systemd/system/netdata.service

SPDX-License-Identifier: GPL-3.0-or-later

[Unit]
Description=Real time performance monitoring

append here other services you want netdata to wait for them to start

After=network.target httpd.service squid.service nfs-server.service mysqld.service mysql.service named.service postfix.service chronyd.service

[Service]
Type=simple
User=netdata
Group=netdata
RuntimeDirectory=netdata
CacheDirectory=netdata
StateDirectory=netdata
LogsDirectory=netdata
RuntimeDirectoryMode=0775
StateDirectoryMode=0755
CacheDirectoryMode=0755
LogsDirectoryMode=2750
EnvironmentFile=-/etc/default/netdata
ExecStart=/usr/sbin/netdata -D $EXTRA_OPTS

saving a big db on slow disks may need some time

TimeoutStopSec=150

restart netdata if it crashes

Restart=on-failure
RestartSec=30

Valid policies: other (the system default) | batch | idle | fifo | rr

To give netdata the max priority, set CPUSchedulingPolicy=rr and CPUSchedulingPriority=99

CPUSchedulingPolicy=batch

This sets the scheduling priority (for policies: rr and fifo).

Priority gets values 1 (lowest) to 99 (highest).

#CPUSchedulingPriority=1

For scheduling policy ‘other’ and ‘batch’, this sets the lowest niceness of netdata (-20 highest to 19 lowest).

Nice=0

[Install]
WantedBy=multi-user.target
root@Test:~#

9

Can you run the tests as user netdata instead of root ?

10

The result is the same. No SSL error as far as I can see.

netdata@Test:~$ curl -I https://api.netdata.cloud/api/v1/env
HTTP/2 404
accept-ranges: bytes
cache-control: no-cache
content-length: 397
content-type: text/html
date: Thu, 22 Dec 2022 16:34:18 GMT
etag: “63a43261-18d”
expires: Thu, 01 Jan 1970 00:00:01 GMT
last-modified: Thu, 22 Dec 2022 10:33:05 GMT
server: nginx/1.17.5
vary: Accept-Encoding
x-frame-options: SAMEORIGIN

netdata@Test:~$ curl https://api.netdata.cloud/api/v1/env
{“errorCode”:“”,“errorMsgKey”:“ErrInvalidClaimID”,“errorMessage”:“claim_id not a valid UUID”,“errorNonRetryable”:true,“errorRetryDelaySeconds”:0}netdata@Test:~$
netdata@Test:~$ wget -S -O - https://api.netdata.cloud/api/v1/env
–2022-12-22 22:04:38-- https://api.netdata.cloud/api/v1/env
Resolving api.netdata.cloud (api.netdata.cloud)… 44.196.50.41, 44.207.131.212, 54.198.178.11
Connecting to api.netdata.cloud (api.netdata.cloud)|44.196.50.41|:443… connected.
HTTP request sent, awaiting response…
HTTP/1.1 400 Bad Request
content-length: 145
content-type: application/json; charset=utf-8
date: Thu, 22 Dec 2022 16:34:39 GMT
vary: Accept-Encoding
2022-12-22 22:04:39 ERROR 400: Bad Request.

netdata@Test:~$

11

Any updates on this?

12

run an update for the ca-certificates package.