ACLK_MAIN ERROR - "Connection Problem" With Proxy

Jonathan_Stampella · October 27, 2023, 6:03pm

I am having a problem connecting to the cloud with my netdata service, I am getting the message “Connection problem”

I have the netdata service set up with docker compose, the server has a proxy for output to the internet so it is configured to output through the proxy and also configured the no_proxy for localhost

version: '3'
services:
  netdata:
    image: netdata/netdata:latest
    container_name: netdata
    hostname: ******.com.ar
    pid: host
    ports:
      - 4000:19999
    restart: unless-stopped
    cap_add:
      - SYS_ADMIN
    security_opt:
      - apparmor:unconfined
    volumes:
      - ./log/:/var/log/netdata/
      - ./netdata.conf:/etc/netdata/netdata.conf
      - netdatalib:/var/lib/netdata
      - netdatacache:/var/cache/netdata
      - /etc/passwd:/host/etc/passwd:ro
      - /etc/group:/host/etc/group:ro
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /etc/os-release:/host/etc/os-release:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
    environment:
      - PGID=979
      - VIRTUALIZATION=vmware
      - TZ=America/Argentina
      - http_proxy=http://*******.com.ar:8080
      - https_proxy=http://*******.com.ar:8080
      - no_proxy=localhost,127.0.0.1,*****.com.ar:4000
      - NETDATA_CLAIM_TOKEN=why9p*****************************************amKQ
      - NETDATA_CLAIM_URL=https://app.netdata.cloud
      - NETDATA_CLAIM_ROOMS=**********************
    networks:
      - network_capacity
volumes:
  netdatalib:
  netdatacache:

networks:
  network_capacity:
    external: true

I comment with **** the confidential data
I clarify that on the host server: I have the exports declared

I don’t know what the problem is due to, I know it is due to the proxy but I don’t know how to solve the problem, the proxy has the connection with the url [app.netdata.cloud] enabled, is another url that netdata uses missing? Is there any extra configuration that must be done in these cases when using proxy? thank you

Manolis_Vasilakis · October 30, 2023, 1:25pm

Hi @Jonathan_Stampella

Is it possible to grep your error.log for any ACLK lines? Should show some more info on the connection status.

Thanks!

Jonathan_Stampella · October 30, 2023, 1:39pm

Hi, in error.log show this

Manolis_Vasilakis · October 30, 2023, 2:23pm

Thanks Jonathan.

Have you followed the steps for defining the proxy here → Connect Agent to Cloud | Learn Netdata ?

Jonathan_Stampella · October 30, 2023, 2:44pm

I set the proxy but I don’t know what else I have to do, I’m lost with the netdata and proxy configuration! run the netdata-kickstart.sh script, first download it to my PC and then paste it on the server because I don’t have the url enabled for https://comunity.netdata.cloud and it gives me this:

Jonathan_Stampella · October 30, 2023, 5:01pm

Could it be because I don’t have that url enabled in the proxy?
Thanks

Jonathan_Stampella · October 30, 2023, 6:46pm

I solved the problem to connect to the cloud, it was the docker image that had an old version of netdata. Now when I go to my profile in netdata I can’t see the connected node

Is there a waiting time until the node is seen online? or do you have to enable something on the server?
or maybe the same error continues and only the interface was updated.

I noticed that in the info it shows me node_id as null

Is there something I’m missing?

# netdata configuration
#
# You can download the latest version of this file, using:
#
#  wget -O /etc/netdata/netdata.conf http://localhost:19999/netdata.conf
# or
#  curl -o /etc/netdata/netdata.conf http://localhost:19999/netdata.conf
#
# You can uncomment and change any of the options below.
# The value shown in the commented settings, is the default value.
#

# global netdata configuration

[global]
	# run as user = netdata
	# host access prefix = /host
	# glibc malloc arena max for plugins = 1
	# cpu cores = 8
	# libuv worker threads = 48
	# hostname = PLVULCAUTOMA
	# enable metric correlations = yes
	# metric correlations method = ks2
	# timezone = America/Argentina
	# OOM score = 0
	# process scheduling policy = batch
	# process nice level = 19
	# pthread stack size = 131072

[db]
	# update every = 1
	# mode = dbengine
	# dbengine page cache size MB = 32
	# dbengine extent cache size MB = 0
	# dbengine enable journal integrity check = no
	# dbengine disk space MB = 256
	# dbengine multihost disk space MB = 256
	# memory deduplication (ksm) = yes
	# cleanup obsolete charts after secs = 3600
	# gap when lost iterations above = 1
	# enable replication = yes
	# seconds to replicate = 86400
	# seconds per replication step = 600
	# cleanup orphan hosts after secs = 3600
	# dbengine use direct io = yes
	# dbengine pages per extent = 64
	# storage tiers = 3
	# dbengine parallel initialization = yes
	# dbengine tier 1 multihost disk space MB = 128
	# dbengine tier 1 update every iterations = 60
	# dbengine tier 1 backfill = new
	# dbengine tier 2 multihost disk space MB = 64
	# dbengine tier 2 update every iterations = 60
	# dbengine tier 2 backfill = new
	# delete obsolete charts files = yes
	# delete orphan hosts files = yes
	# enable zero metrics = no
	# replication threads = 1

[directories]
	# config = /etc/netdata
	# stock config = /usr/lib/netdata/conf.d
	# log = /var/log/netdata
	# web = /usr/share/netdata/web
	# cache = /var/cache/netdata
	# lib = /var/lib/netdata
	# home = /root
	# lock = /var/lib/netdata/lock
	# plugins = "/usr/libexec/netdata/plugins.d" "/etc/netdata/custom-plugins.d"
	# registry = /var/lib/netdata/registry
	# stock health config = /usr/lib/netdata/conf.d/health.d
	# health config = /etc/netdata/health.d

[logs]
	# debug flags = 0x0000000000000000
	# debug = /var/log/netdata/debug.log
	# error = /var/log/netdata/error.log
	# collector = /var/log/netdata/collector.log
	# access = /var/log/netdata/access.log
	# health = /var/log/netdata/health.log
	# facility = daemon
	# errors flood protection period = 1200
	# errors to trigger flood protection = 200
	# severity level = info

[environment variables]
	# PATH = /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin
	# PYTHONPATH = 

[host labels]
	# name = value

[sqlite]
	# auto vacuum = INCREMENTAL
	# synchronous = NORMAL
	# journal mode = WAL
	# temp store = MEMORY
	# journal size limit = 16777216
	# cache size = -2000

[cloud]
	proxy = http://************.com.ar:8080
	# conversation log = no
	# query thread count = 4
	# statistics = yes

[ml]
	# enabled = yes
	# maximum num samples to train = 21600
	# minimum num samples to train = 900
	# train every = 10800
	# number of models per dimension = 18
	# delete models older than = 604800
	# num samples to diff = 1
	# num samples to smooth = 3
	# num samples to lag = 5
	# random sampling ratio = 0.20000
	# maximum number of k-means iterations = 1000
	# dimension anomaly score threshold = 0.99000
	# host anomaly rate threshold = 1.00000
	# anomaly detection grouping method = average
	# anomaly detection grouping duration = 300
	# num training threads = 4
	# flush models batch size = 128
	# dimension anomaly rate suppression window = 900
	# dimension anomaly rate suppression threshold = 450
	# enable statistics charts = yes
	# hosts to skip from training = !*
	# charts to skip from training = netdata.*
	# stream anomaly detection charts = yes

[health]
	# silencers file = /var/lib/netdata/health.silencers.json
	# enabled = yes
	# is ephemeral = no
	# has unstable connection = no
	# run at least every seconds = 10
	# postpone alarms during hibernation for seconds = 60
	# default repeat warning = never
	# default repeat critical = never
	# in memory max health log entries = 1000
	# health log history = 432000
	# enabled alarms = *
	# script to execute on alarm = /usr/libexec/netdata/plugins.d/alarm-notify.sh
	# use summary for notifications = yes
	# enable stock health configuration = yes

[web]
	# default port = 19999
	# ssl key = /etc/netdata/ssl/key.pem
	# ssl certificate = /etc/netdata/ssl/cert.pem
	# tls version = 1.3
	# tls ciphers = none
	# ses max tg_des_window = 15
	# des max tg_des_window = 15
	# mode = static-threaded
	# listen backlog = 4096
	# bind to = *
	# disconnect idle clients after seconds = 60
	# timeout for first request = 60
	# accept a streaming request every seconds = 0
	# respect do not track policy = no
	# x-frame-options response header = 
	# allow connections from = localhost *
	# allow connections by dns = heuristic
	# allow dashboard from = localhost *
	# allow dashboard by dns = heuristic
	# allow badges from = *
	# allow badges by dns = heuristic
	# allow streaming from = *
	# allow streaming by dns = heuristic
	# allow netdata.conf from = localhost fd* 10.* 192.168.* 172.16.* 172.17.* 172.18.* 172.19.* 172.20.* 172.21.* 172.22.* 172.23.* 172.24.* 172.25.* 172.26.* 172.27.* 172.28.* 172.29.* 172.30.* 172.31.* UNKNOWN
	# allow netdata.conf by dns = no
	# allow management from = localhost
	# allow management by dns = heuristic
	# enable gzip compression = yes
	# gzip compression strategy = default
	# gzip compression level = 3
	# ssl skip certificate verification = no
	# web server threads = 6
	# web server max sockets = 262144

[registry]
	enabled = yes
	# netdata unique id file = /var/lib/netdata/registry/netdata.public.unique.id
	# registry db file = /var/lib/netdata/registry/registry.db
	# registry log file = /var/lib/netdata/registry/registry-log.db
	# registry save db every new entries = 1000000
	# registry expire idle persons days = 365
	# registry domain = 
	# registry to announce = https://registry.my-netdata.io
	# registry hostname = PLVULCAUTOMA
	# verify browser cookies support = yes
	# enable cookies SameSite and Secure = yes
	# max URL length = 1024
	# max URL name length = 50
	# use mmap = no
	# netdata management api key file = /var/lib/netdata/netdata.api.key
	# allow from = *
	# allow by dns = heuristic

[global statistics]
	# update every = 1

inside the netdata container it has a connection with app.netdata

Manolis_Vasilakis · October 31, 2023, 1:10pm

It appears that the connection still doesn’t work…

Can you post the output of http://localhost:19999/api/v1/aclk? Can you check again the error.log for ACLK entries?

Jonathan_Stampella · October 31, 2023, 2:57pm

api/v1/aclk

{
  "aclk-available": true,
  "aclk-version": 2,
  "protocols-supported": [
    "Protobuf"
  ],
  "agent-claimed": true,
  "claimed-id": "8a8a3fd9-3325-40d9-8d36-cce6d95be610",
  "cloud-url": "https://app.netdata.cloud",
  "online": false,
  "used-cloud-protocol": "Protobuf",
  "mqtt-version": 5,
  "received-app-layer-msgs": 0,
  "received-mqtt-pubacks": 0,
  "reconnect-count": 0,
  "last-connect-time-utc": null,
  "last-connect-time-puback-utc": null,
  "last-disconnect-time-utc": null,
  "next-connection-attempt-utc": "2023-10-31 14:54:08",
  "last-backoff-value": 164.85499572753906,
  "banned-by-cloud": false,
  "node-instances": [
    {
      "hostname": "PLVULCAUTOMA",
      "mguid": "8a8a3fd9-3325-40d9-8d36-cce6d95be610",
      "claimed_id": "8a8a3fd9-3325-40d9-8d36-cce6d95be610",
      "node-id": null,
      "streaming-hops": 0,
      "relationship": "self",
      "streaming-online": true,
      "alert-sync-status": {
        
      }
    }
  ]
}

file error.log

2023-10-31 14:49:02: netdata INFO  : TRAIN[0] : thread created with task id 41842
2023-10-31 14:49:02: netdata INFO  : TRAIN[0] : set name of thread 41842 to TRAIN[0]
2023-10-31 14:49:02: netdata INFO  : P[diskspace slo : set name of thread 41841 to P[diskspace slo
2023-10-31 14:49:02: netdata INFO  : TRAIN[1] : thread created with task id 41843
2023-10-31 14:49:02: netdata INFO  : TRAIN[1] : set name of thread 41843 to TRAIN[1]
2023-10-31 14:49:02: netdata ERROR : STATSD_FLUSH : LISTENER: IPv6 bind() on ip '::1' port 8125, socktype 1 failed. (errno 99, Address not available)
2023-10-31 14:49:02: netdata ERROR : STATSD_FLUSH : LISTENER: Cannot bind to ip '::1', port 8125
2023-10-31 14:49:02: netdata INFO  : STATSD_FLUSH : LISTENER: Listen socket udp:127.0.0.1:8125 opened successfully.
2023-10-31 14:49:02: netdata INFO  : STATSD_FLUSH : LISTENER: Listen socket tcp:127.0.0.1:8125 opened successfully.
2023-10-31 14:49:02: netdata INFO  : WEB[6] : POLLFD: LISTENER: listening on 'tcp:[::]:19999'
2023-10-31 14:49:02: netdata INFO  : PREDICT : thread created with task id 41837
2023-10-31 14:49:02: netdata INFO  : PREDICT : set name of thread 41837 to PREDICT
2023-10-31 14:49:02: netdata INFO  : WEB[5] : set name of thread 41830 to WEB[5]
2023-10-31 14:49:02: netdata INFO  : P[proc] : set name of thread 41829 to P[proc]
2023-10-31 14:49:02: netdata INFO  : P[proc netdev] : thread created with task id 41846
2023-10-31 14:49:02: netdata INFO  : P[proc netdev] : set name of thread 41846 to P[proc netdev]
2023-10-31 14:49:02: netdata INFO  : WEB[5] : POLLFD: LISTENER: listening on 'tcp:0.0.0.0:19999'
2023-10-31 14:49:02: netdata INFO  : WEB[5] : POLLFD: LISTENER: listening on 'tcp:[::]:19999'
2023-10-31 14:49:02: netdata INFO  : STATSD_IN[1] : thread created with task id 41845
2023-10-31 14:49:02: netdata INFO  : STATSD_IN[1] : set name of thread 41845 to STATSD_IN[1]
2023-10-31 14:49:02: netdata INFO  : STATSD_IN[1] : POLLFD: LISTENER: listening on 'udp:127.0.0.1:8125'
2023-10-31 14:49:02: netdata INFO  : STATSD_IN[1] : POLLFD: LISTENER: listening on 'tcp:127.0.0.1:8125'
2023-10-31 14:49:02: netdata INFO  : PD[ioping] : PLUGINSD: 'host:PLVULCAUTOMA' connected to '/usr/libexec/netdata/plugins.d/ioping.plugin' running on pid 41844
2023-10-31 14:49:02: netdata INFO  : TRAIN[2] : thread created with task id 41848
2023-10-31 14:49:02: netdata INFO  : TRAIN[2] : set name of thread 41848 to TRAIN[2]
2023-10-31 14:49:02: netdata INFO  : MAIN : NETDATA STARTUP: in       5 ms, start the static threads - next: initialize commands API
2023-10-31 14:49:02: netdata INFO  : MAIN : Initializing command server.
2023-10-31 14:49:02: netdata INFO  : PD[go.d] : PLUGINSD: 'host:PLVULCAUTOMA' connected to '/usr/libexec/netdata/plugins.d/go.d.plugin' running on pid 41850
2023-10-31 14:49:02: netdata INFO  : TRAIN[3] : thread created with task id 41851
2023-10-31 14:49:02: netdata INFO  : TRAIN[3] : set name of thread 41851 to TRAIN[3]
2023-10-31 14:49:02: netdata INFO  : MAIN : NETDATA STARTUP: in       0 ms, initialize commands API - next: ready
2023-10-31 14:49:02: netdata INFO  : MAIN : NETDATA STARTUP: completed in 423 ms. Enjoy real-time performance monitoring!
2023-10-31 14:49:02: netdata INFO  : MAIN : /usr/libexec/netdata/plugins.d/anonymous-statistics.sh 'START' '-' '-'
2023-10-31 14:49:02: netdata INFO  : PD[apps] : PLUGINSD: 'host:PLVULCAUTOMA' connected to '/usr/libexec/netdata/plugins.d/apps.plugin' running on pid 41854
2023-10-31 14:49:02: netdata INFO  : ACLK_MAIN : Wait before attempting to reconnect in 0.000 seconds
2023-10-31 14:49:02: netdata INFO  : ACLK_MAIN : Attempting connection now
2023-10-31 14:49:02: netdata INFO  : ACLK_STATS : thread created with task id 41857
2023-10-31 14:49:02: netdata INFO  : ACLK_STATS : set name of thread 41857 to ACLK_STATS
2023-10-31 14:49:02: netdata INFO  : PD[perf] : PLUGINSD: 'host:PLVULCAUTOMA' connected to '/usr/libexec/netdata/plugins.d/perf.plugin' running on pid 41856
2023-10-31 14:49:02: netdata INFO  : PD[perf] : PLUGINSD: plugin called DISABLE. Disabling it.
2023-10-31 14:49:02: netdata INFO  : PD[perf] : PLUGINSD: 'host:PLVULCAUTOMA', '/usr/libexec/netdata/plugins.d/perf.plugin' (pid 41856) disconnected after 0 successful data collections (ENDs).
2023-10-31 14:49:02: netdata ERROR : PD[perf] : child pid 41856 exited with code 1.
2023-10-31 14:49:02: netdata ERROR : PD[perf] : PLUGINSD: 'host:PLVULCAUTOMA', '/usr/libexec/netdata/plugins.d/perf.plugin' (pid 41856) exited with error code 1 and haven't collected any data. Disabling it. (errno 22, Invalid argument)
2023-10-31 14:49:02: netdata INFO  : PD[perf] : thread with task id 41838 finished
2023-10-31 14:49:02: netdata INFO  : PD[ioping] : PLUGINSD: plugin called DISABLE. Disabling it.
2023-10-31 14:49:02: netdata INFO  : PD[ioping] : PLUGINSD: 'host:PLVULCAUTOMA', '/usr/libexec/netdata/plugins.d/ioping.plugin' (pid 41844) disconnected after 0 successful data collections (ENDs).
2023-10-31 14:49:02: netdata INFO  : PD[python.d] : PLUGINSD: 'host:PLVULCAUTOMA' connected to '/usr/libexec/netdata/plugins.d/python.d.plugin' running on pid 41889
2023-10-31 14:49:02: netdata INFO  : PD[debugfs] : PLUGINSD: 'host:PLVULCAUTOMA' connected to '/usr/libexec/netdata/plugins.d/debugfs.plugin' running on pid 41877
2023-10-31 14:49:02: netdata ERROR : PD[ioping] : child pid 41844 exited with code 1.
2023-10-31 14:49:02: netdata ERROR : PD[ioping] : PLUGINSD: 'host:PLVULCAUTOMA', '/usr/libexec/netdata/plugins.d/ioping.plugin' (pid 41844) exited with error code 1 and haven't collected any data. Disabling it. (errno 22, Invalid argument)
2023-10-31 14:49:02: netdata INFO  : PD[ioping] : thread with task id 41836 finished
2023-10-31 14:49:02: netdata ERROR : P[cgroups] : child pid 41893 exited with code 127.
2023-10-31 14:49:02: netdata ERROR : P[cgroups] : child pid 41896 exited with code 127.
2023-10-31 14:49:02: netdata INFO  : PD[charts.d] : PLUGINSD: plugin called DISABLE. Disabling it.
2023-10-31 14:49:02: netdata INFO  : PD[charts.d] : PLUGINSD: 'host:PLVULCAUTOMA', '/usr/libexec/netdata/plugins.d/charts.d.plugin' (pid 41840) disconnected after 0 successful data collections (ENDs).
2023-10-31 14:49:02: netdata INFO  : PD[charts.d] : PLUGINSD: 'host:PLVULCAUTOMA', '/usr/libexec/netdata/plugins.d/charts.d.plugin' (pid 41840) does not generate useful output but it reports success (exits with 0). Will not start it again - it is now disabled..
2023-10-31 14:49:02: netdata ERROR : ACLK_MAIN : Expected response to start with "HTTP/1.1 " (errno 11, Resource temporarily unavailable)
2023-10-31 14:49:02: netdata ERROR : ACLK_MAIN : Error parsing HTTP response
2023-10-31 14:49:02: netdata ERROR : ACLK_MAIN : Error reading or parsing response from server (errno 22, Invalid argument)
2023-10-31 14:49:02: netdata ERROR : ACLK_MAIN : Failed to CONNECT with proxy
2023-10-31 14:49:02: netdata ERROR : ACLK_MAIN : Error trying to contact env endpoint (errno 22, Invalid argument)
2023-10-31 14:49:02: netdata ERROR : ACLK_MAIN : Failed to Get ACLK environment (cannot contact ENV endpoint)
2023-10-31 14:49:02: netdata INFO  : ACLK_MAIN : Wait before attempting to reconnect in 1.431 seconds
2023-10-31 14:49:03: netdata INFO  : MAIN : /usr/libexec/netdata/plugins.d/anonymous-statistics.sh 'INCOMPLETE_SHUTDOWN' '-' '-'
2023-10-31 14:49:03: netdata INFO  : P[proc] : Using now_boottime_usec() for uptime (dt is 9 ms)
2023-10-31 14:49:03: netdata LOG FLOOD PROTECTION too many logs (201 logs in 1 seconds, threshold is set to 200 logs in 1200 seconds). Preventing more logs from process 'netdata' for 1199 seconds.

I can’t figure out what is failing, what DNS application are they pointing to? apart from app.netdata.cloud does it point to another?

underhood · November 2, 2023, 8:22am

Any information about the proxy server and it’s configuration? What is the actual proxy server SQUID, ntlm?

Maybe I can add log there
2023-10-31 14:49:02: netdata ERROR : ACLK_MAIN : Expected response to start with "HTTP/1.1 " (errno 11, Resource temporarily unavailable)
to actually print what we received instead which could give us some hints.

Currently I can just see the proxy doesnt reply with HTTP/1.1 as expected but can’t tell why.

Jonathan_Stampella · November 3, 2023, 12:30pm

The type of proxy we use is “IP-based authentication proxy”

with telnet not working

I have enabled what is port 80 and 443 of the url app.netdata.cloud, is it only that url to which the netdata app makes requests and those ports?
Thanks

Jonathan_Stampella · November 3, 2023, 4:02pm

I attach more details of the system

##########################################
netdata -W buildinfo

Packaging:
    Netdata Version ____________________________________________ : v1.43.2
    Installation Type __________________________________________ : oci
    Package Architecture _______________________________________ : x86_64
    Package Distro _____________________________________________ : unknown
    Configure Options __________________________________________ :  '--prefix=/usr' '--sysconfdir=/etc' '--localstatedir=/var' '--libexecdir=/usr/libexec' '--libdir=/usr/lib' '--with-math' '--with-user=netdata' '--without-bundled-protobuf' '--disable-dependency-tracking' '--enable-lto' '--disable-ebpf' 'CFLAGS=-ffunction-sections -fdata-sections -O2 -funroll-loops -pipe' 'LDFLAGS=-Wl,--gc-sections'
Default Directories:
    User Configurations ________________________________________ : /etc/netdata
    Stock Configurations _______________________________________ : /usr/lib/netdata/conf.d
    Ephemeral Databases (metrics data, metadata) _______________ : /var/cache/netdata
    Permanent Databases ________________________________________ : /var/lib/netdata
    Plugins ____________________________________________________ : /usr/libexec/netdata/plugins.d
    Static Web Files ___________________________________________ : /usr/share/netdata/web
    Log Files __________________________________________________ : /var/log/netdata
    Lock Files _________________________________________________ : /var/lib/netdata/lock
    Home _______________________________________________________ : /var/lib/netdata
Operating System:
    Kernel _____________________________________________________ : Linux
    Kernel Version _____________________________________________ : 4.18.0-348.12.2.el8_5.x86_64
    Operating System ___________________________________________ : Red Hat Enterprise Linux
    Operating System ID ________________________________________ : rhel
    Operating System ID Like ___________________________________ : fedora
    Operating System Version ___________________________________ : 8.5 (Ootpa)
    Operating System Version ID ________________________________ : 3.18.4
    Detection __________________________________________________ : /host/etc/os-release
Hardware:
    CPU Cores __________________________________________________ : 8
    CPU Frequency ______________________________________________ : 3000000000
    CPU Architecture ___________________________________________ : 67240763392
    RAM Bytes __________________________________________________ : 1428076625920
    Disk Capacity ______________________________________________ : x86_64
    Virtualization Technology __________________________________ : vmware
    Virtualization Detection ___________________________________ : provided
Container:
    Container __________________________________________________ : docker
    Container Detection ________________________________________ : dockerenv
    Container Orchestrator _____________________________________ : none
    Container Operating System _________________________________ : Alpine Linux
    Container Operating System ID ______________________________ : alpine
    Container Operating System ID Like _________________________ : unknown
    Container Operating System Version _________________________ : unknown
    Container Operating System Version ID ______________________ : 3.18.4
    Container Operating System Detection _______________________ : /etc/os-release
Features:
    Built For __________________________________________________ : Linux
    Netdata Cloud ______________________________________________ : YES
    Health (trigger alerts and send notifications) _____________ : YES
    Streaming (stream metrics to parent Netdata servers) _______ : YES
    Replication (fill the gaps of parent Netdata servers) ______ : YES
    Streaming and Replication Compression ______________________ : YES (lz4)
    Contexts (index all active and archived metrics) ___________ : YES
    Tiering (multiple dbs with different metrics resolution) ___ : YES (5)
    Machine Learning ___________________________________________ : YES
Database Engines:
    dbengine ___________________________________________________ : YES
    alloc ______________________________________________________ : YES
    ram ________________________________________________________ : YES
    map ________________________________________________________ : YES
    save _______________________________________________________ : YES
    none _______________________________________________________ : YES
Connectivity Capabilities:
    ACLK (Agent-Cloud Link: MQTT over WebSockets over TLS) _____ : YES
    static (Netdata internal web server) _______________________ : YES
    h2o (web server) ___________________________________________ : NO
    WebRTC (experimental) ______________________________________ : NO
    Native HTTPS (TLS Support) _________________________________ : YES
    TLS Host Verification ______________________________________ : YES
Libraries:
    LZ4 (extremely fast lossless compression algorithm) ________ : YES
    zlib (lossless data-compression library) ___________________ : YES
    Judy (high-performance dynamic arrays and hashtables) ______ : YES (bundled)
    dlib (robust machine learning toolkit) _____________________ : YES (bundled)
    protobuf (platform-neutral data serialization protocol) ____ : YES (system)
    OpenSSL (cryptography) _____________________________________ : YES
    libdatachannel (stand-alone WebRTC data channels) __________ : NO
    JSON-C (lightweight JSON manipulation) _____________________ : YES
    libcap (Linux capabilities system operations) ______________ : NO
    libcrypto (cryptographic functions) ________________________ : YES
    libm (mathematical functions) ______________________________ : YES
    jemalloc ___________________________________________________ : NO
    TCMalloc ___________________________________________________ : NO
Plugins:
    apps (monitor processes) ___________________________________ : YES
    cgroups (monitor containers and VMs) _______________________ : YES
    cgroup-network (associate interfaces to CGROUPS) ___________ : YES
    proc (monitor Linux systems) _______________________________ : YES
    tc (monitor Linux network QoS) _____________________________ : YES
    diskspace (monitor Linux mount points) _____________________ : YES
    freebsd (monitor FreeBSD systems) __________________________ : NO
    macos (monitor MacOS systems) ______________________________ : NO
    statsd (collect custom application metrics) ________________ : YES
    timex (check system clock synchronization) _________________ : YES
    idlejitter (check system latency and jitter) _______________ : YES
    bash (support shell data collection jobs - charts.d) _______ : YES
    debugfs (kernel debugging metrics) _________________________ : YES
    cups (monitor printers and print jobs) _____________________ : NO
    ebpf (monitor system calls) ________________________________ : NO
    freeipmi (monitor enterprise server H/W) ___________________ : YES
    nfacct (gather netfilter accounting) _______________________ : NO
    perf (collect kernel performance events) ___________________ : YES
    slabinfo (monitor kernel object caching) ___________________ : YES
    Xen ________________________________________________________ : NO
    Xen VBD Error Tracking _____________________________________ : NO
Exporters:
    AWS Kinesis ________________________________________________ : NO
    GCP PubSub _________________________________________________ : NO
    MongoDB ____________________________________________________ : YES
    Prometheus (OpenMetrics) Exporter __________________________ : YES
    Prometheus Remote Write ____________________________________ : YES
    Graphite ___________________________________________________ : YES
    Graphite HTTP / HTTPS ______________________________________ : YES
    JSON _______________________________________________________ : YES
    JSON HTTP / HTTPS __________________________________________ : YES
    OpenTSDB ___________________________________________________ : YES
    OpenTSDB HTTP / HTTPS ______________________________________ : YES
    All Metrics API ____________________________________________ : YES
    Shell (use metrics in shell scripts) _______________________ : YES
Debug/Developer Features:
    Trace All Netdata Allocations (with charts) ________________ : NO
    Developer Mode (more runtime checks, slower) _______________ : NO

I did a test by removing the proxy from the netdata.conf file. and I get these errors:

2023-11-03 17:28:20: netdata ERROR : ACLK_MAIN : Failed to connect to '54.198.178.11', port '443' (errno 111, Connection refused)
2023-11-03 17:28:20: netdata ERROR : ACLK_MAIN : Failed to connect to '44.207.131.212', port '443' (errno 111, Connection refused)
2023-11-03 17:28:20: netdata ERROR : ACLK_MAIN : Failed to connect to '44.196.50.41', port '443' (errno 111, Connection refused)
2023-11-03 17:28:20: netdata ERROR : ACLK_MAIN : Error connecting TCP socket to "app.netdata.cloud"
2023-11-03 17:28:20: netdata ERROR : ACLK_MAIN : Error trying to contact env endpoint (errno 22, Invalid argument)
2023-11-03 17:28:20: netdata ERROR : ACLK_MAIN : Failed to Get ACLK environment (cannot contact ENV endpoint)
2023-11-03 17:28:20: netdata INFO  : ACLK_MAIN : Wait before attempting to reconnect in 1.714 seconds

but if I put it in the .conf file
[nube]
proxy = http://************:8080

2023-11-03 17:31:31: netdata ERROR : ACLK_MAIN : Expected response to start with "HTTP/1.1 " (errno 11, Resource temporarily unavailable)
2023-11-03 17:31:31: netdata ERROR : ACLK_MAIN : Error parsing HTTP response
2023-11-03 17:31:31: netdata ERROR : ACLK_MAIN : Error reading or parsing response from server (errno 22, Invalid argument)
2023-11-03 17:31:31: netdata ERROR : ACLK_MAIN : Failed to CONNECT with proxy
2023-11-03 17:31:31: netdata ERROR : ACLK_MAIN : Error trying to contact env endpoint (errno 22, Invalid argument)
2023-11-03 17:31:31: netdata ERROR : ACLK_MAIN : Failed to Get ACLK environment (cannot contact ENV endpoint)
2023-11-03 17:31:31: netdata INFO  : ACLK_MAIN : Wait before attempting to reconnect in 1.564 seconds

Jonathan_Stampella · December 19, 2023, 3:32pm

something new?
thank you

ars · May 14, 2024, 9:13pm

Same problem.

Sniffer shows HTTP CONNECT message without “Host” header. Nginx proxy reply 400 “Bad Request”.

Netdata agent MUST include Host header in CONNECT request message.

Fix please.

ilyam8 · May 16, 2024, 6:49am

@ars hey, thanks. Fixed in Include the Host in the HTTP header by stelfrag · Pull Request #17670 · netdata/netdata · GitHub.

ars · May 22, 2024, 10:31am

@ilyam8 hey, another problem after checks v1.45.5

CONNECT to app.netdata.cloud is correct now. Thanks!
But CONNECT to mqtt.netdata.cloud not fixed

hugo · May 22, 2024, 12:18pm

Thanks for reporting it @ars , we confirm the issue and are working on a fix Include the Host in the HTTP header (mqtt) by stelfrag · Pull Request #17731 · netdata/netdata · GitHub

hugo · May 23, 2024, 8:15am

The mentioned PR is merged and the fix is available on the latest nightly release.

ars · June 7, 2024, 9:38am

Thanks! Works great!

Topic		Replies	Views
Netdata agent is unable to connect with netdata cloud Help	11	1481	October 23, 2023
Failed to connect to https://app.netdata.cloud, return code 6 Help	4	955	October 14, 2022
Netdata 1.30.0 Agents fail to connect to Cloud Help cloud	8	1141	April 19, 2021
Netdata container and disabling cloud connectivity Help cloud	5	3109	December 16, 2020
Most our hosts can stream to the cloud Help cloud	2	612	February 2, 2022

ACLK_MAIN ERROR - "Connection Problem" With Proxy

Related topics