Unable to get parent/child streaming working

[tsl]

Problem/Question

It’s the first time I installed netdata - I wanted to test it as it seems great on paper. However, while I can successfully put a parent online and access its data from https://xxx.xxx.xxx.xxx:19999, there is no way I can get streaming working. I set up two different machines as child, but I always get the error "“API KEY DISABLED PERMISSION DENIED” and “API KEY DISABLED PERMISSION DENIED”. Like the API key was wrong.

This is a little excerpt from the logs:

time=2024-07-01T21:02:30.737+02:00 comm=netdata source=access level=debug tid=166614 thread=WEB[3] src_transport=http src_ip=192.XXX.X.XXX src_port=43704 conn=0 msg="[192.168.2.189]:43704 CONNECTED"
time=2024-07-01T21:02:30.742+02:00 comm=netdata source=access level=warning tid=166614 thread=WEB[3] msg_id=ed4cdb8f1beb4ad3b57cb3cae2d162fa node=XXXXXXXXXXXXXX src_transport=http role=none permissions=0x0 src_i
p=192.168.2.189 src_port=43704 req_method=GET code="API KEY DISABLED PERMISSION DENIED" conn=0 transaction=1c52117400ed42008b90f17595beef63 request="key=xxxxxxxxxxxxxxxxxxxxxxxxx&hostname=XXXXXXXXXX&registry_hostname=XXXXXXXXXXX&machine_guid=e9facdea-371c-11ef-81bc-dca632c7c21c&update_every=1&os=linux&timezone=Europe/Madrid&abbrev_timezone=CEST&utc_offset=7200&hops=1&ml_capable=1&ml_enabled=1&mc_versi
on=1&ver=14548984&NETDATA_INSTANCE_CLOUD_TYPE=unknown&NETDATA_INSTANCE_CLOUD_INSTANCE_TYPE=unknown&NETDATA_INSTANCE_CLOUD_INSTANCE_REGION=unknown&NETDATA_SYSTEM_OS_NAME=Debian+GNU%2fLinux&NETDATA_SYSTEM_OS_ID=d
ebian&NETDATA_SYSTEM_OS_ID_LIKE=unknown&NETDATA_SYSTEM_OS_VERSION=11+%28bullseye%29&NETDATA_SYSTEM_OS_VERSION_ID=11&NETDATA_SYSTEM_OS_DETECTION=/etc/os-release&NETDATA_HOST_IS_K8S_NODE=false&NETDATA_SYSTEM_KERN
EL_NAME=Linux&NETDATA_SYSTEM_KERNEL_VERSION=5.10.103-v8%2b&NETDATA_SYSTEM_ARCHITECTURE=aarch64&NETDATA_SYSTEM_VIRTUALIZATION=none&NETDATA_SYSTEM_VIRT_DETECTION=systemd-detect-virt&NETDATA_SYSTEM_CONTAINER=none&
NETDATA_SYSTEM_CONTAINER_DETECTION=systemd-detect-virt&NETDATA_CONTAINER_OS_NAME=none&NETDATA_CONTAINER_OS_ID=none&NETDATA_CONTAINER_OS_ID_LIKE=none&NETDATA_CONTAINER_OS_VERSION=none&NETDATA_CONTAINER_OS_VERSIO
N_ID=none&NETDATA_CONTAINER_OS_DETECTION=none&NETDATA_SYSTEM_CPU_LOGICAL_CPU_COUNT=4&NETDATA_SYSTEM_CPU_FREQ=1500000000&NETDATA_SYSTEM_TOTAL_RAM=8192532480&NETDATA_SYSTEM_TOTAL_DISK_SIZE=255988948992&NETDATA_PR
OTOCOL_VERSION=1.1" msg="api_key:'xxxxxxxxxxxxxxxxxxxxxxxxxxx' machine_guid:'e9facdea-371c-11ef-81bc-dca632c7c21c' msg:'API key is not enabled'"

time=2024-07-01T21:02:30.742+02:00 comm=netdata source=access level=debug tid=166614 thread=WEB[3] src_transport=http src_ip=192.XXX.X.XXX src_port=43704 conn=0 msg="
[192.168.2.189]:43704 DISCONNECTED"
time=2024-07-01T21:02:30.742+02:00 comm=netdata source=access level=warning tid=166614 thread=WEB[3] role=none permissions=0x0 src_ip=192.XXX.X.XXX src_port=43704 req_method=STREAM code=401 conn=0 transaction=1
c52117400ed42008b90f17595beef63 sent_bytes=67 size_bytes=67 prep_ut=0 sent_ut=0 total_ut=442 request="key=xxxxxxxxxxxxxxxxxxxxx&hostname=xxxxxxxxxxx&registry_hostname=xxxxxxxxxxx&machine_guid
=e9facdea-371c-11ef-81bc-dca632c7c21c&update_every=1&os=linux&timezone=Europe/Madrid&abbrev_timezone=CEST&utc_offset=7200&hops=1&ml_capable=1&ml_enabled=1&mc_version=1&ver=14548984&NETDATA_INSTANCE_CLOUD_TYPE=u
nknown&NETDATA_INSTANCE_CLOUD_INSTANCE_TYPE=unknown&NETDATA_INSTANCE_CLOUD_INSTANCE_REGION=unknown&NETDATA_SYSTEM_OS_NAME=Debian+GNU%2fLinux&NETDATA_SYSTEM_OS_ID=debian&NETDATA_SYSTEM_OS_ID_LIKE=unknown&NETDATA
_SYSTEM_OS_VERSION=11+%28bullseye%29&NETDATA_SYSTEM_OS_VERSION_ID=11&NETDATA_SYSTEM_OS_DETECTION=/etc/os-release&NETDATA_HOST_IS_K8S_NODE=false&NETDATA_SYSTEM_KERNEL_NAME=Linux&NETDATA_SYSTEM_KERNEL_VERSION=5.1
0.103-v8%2b&NETDATA_SYSTEM_ARCHITECTURE=aarch64&NETDATA_SYSTEM_VIRTUALIZATION=none&NETDATA_SYSTEM_VIRT_DETECTION=systemd-detect-virt&NETDATA_SYSTEM_CONTAINER=none&NETDATA_SYSTEM_CONTAINER_DETECTION=systemd-dete
ct-virt&NETDATA_CONTAINER_OS_NAME=none&NETDATA_CONTAINER_OS_ID=none&NETDATA_CONTAINER_OS_ID_LIKE=none&NETDATA_CONTAINER_OS_VERSION=none&NETDATA_CONTAINER_OS_VERSION_ID=none&NETDATA_CONTAINER_OS_DETECTION=none&N
ETDATA_SYSTEM_CPU_LOGICAL_CPU_COUNT=4&NETDATA_SYSTEM_CPU_FREQ=1500000000&NETDATA_SYSTEM_TOTAL_RAM=8192532480&NETDATA_SYSTEM_TOTAL_DISK_SIZE=255988948992&NETDATA_PROTOCOL_VERSION=1.1"

Relevant docs you followed/actions you took to solve the issue:

Netdata Agent Configuration | Learn Netdata

Configuring Metrics Centralization Points | Learn Netdata

Need to create a parent and child stream

I have two virtual machine one is where our ML models will run in that virtual machine we have installed Netdata as a client consider this as a child. And on other virtual machine we have installed netdata now this virtual machine need to be as a central monitor or parent . i have configured the netdata.conf file as shown below Parent netdata.conf file:- [global] run as user = netdata # default storage size - increase for longer data retention page cache size = 32 dbengine multihost disk s…

Environment/Browser/Agent’s version etc

Parent:
Raspberry PI 4 - 4Gb RAM
OS: Debian Bullseye
Netdata version: netdata v1.46.0-68-nightly (installed via Kickstarter first and then I tried with Debian’s packages)

netdata.conf - all standard values except:

[web]
         allow connections from = *

[registry]
        enabled = yes
        registry to announce = http://192.168.2.11:19999

---

stream.conf - all standard values except:

[11111111-2222-3333-4444-555555555555]
    type = api
    enabled = yes
    allow from = *
    default postpone alarms on connect seconds = 60

---

Child 1:
Raspberry PI 4 - 8Gb RAM
OS: Debian Bullseye
Netdata version: netdata v1.46.0-68-nightly

netdata.conf - all standard values

stream.conf - all standard values except:

[stream]
    enabled = yes
    destination = 192.XXX.X.XX
    api key = 11111111-2222-3333-4444-555555555555
    timeout seconds = 60
    default port = 19999
    send charts matching = *
    buffer size bytes = 1048576
    reconnect delay seconds = 5
    initial clock resync iterations = 60

---

Child 2:
Raspberry PI 1 - 256Mb
OS: Debian Bullseye
Netdata version: netdata v1.46.0-34-g2ea4979ec (compiled manually as any other method produced an executable that would fail with “illegal instruction”.

netdata.conf - all standard values except:

[global]
        run as user = netdata
        web files owner = root
        web files group = root
[web]
        bind to = *

---

stream.conf - all standard values except:

[stream]
    enabled = yes
    destination = 192.XXX.X.XX
    api key = 11111111-2222-3333-4444-555555555555
    timeout seconds = 60
    default port = 19999
    send charts matching = *
    buffer size bytes = 10485760
    reconnect delay seconds = 5
    initial clock resync iterations = 60

What I expected to happen

Streaming working properly, with API accepted

[ilyam8]

Hi. Can you please share your Parent and Children stream.conf?

• Redact API key.
• Please use preformatted text for stream.conf.

This is how to format multiline text (do the same but remove \)

\```
TEXT
\```

---

but I always get the error "“API KEY DISABLED PERMISSION DENIED” and “API KEY DISABLED PERMISSION DENIED”

It means that you didn’t really follow Configuring Metrics Centralization Points. Pay attention to Configuring Parent/Children sections.

[tsl]

Apologies, I did published the query before I finished pasting the relevant data. The post is now updated. I have looked at the Configuring Metrics Centralization Points, but could not see anything that I could have missed. Of course I used uuidgen to generate the API and tried with three different ones, just in case. Connectivity happens as the server sees the data coming in, but I cannot understand why is not allowing the childs to go through…

Thank you very much for your help!

[tsl]

And I’ve updated the main post with the code formatting, I hope it’s easier to read now.
Thank you.

[ilyam8]

You either

• didn’t restart Parent after modifying stream.conf or
• the API key is different on Parent and Child instances

I can ensure you that streaming works 100%. The issue is on your side.

[tsl]

The configuration page is quite clear, this is why I’m asking here. There is little that I see that can go wrong, even if it’s the first time I play with Netdata.

Parent and children have been restarted many times. This morning I even reinstalled one of the children and the parent, no change.

The API is absokutely the same on all three machines, as I literally copy and pasted them in the terminal from uuidgen’s output.

Is there any way to see what the parent is receiving or maybe simulating an handshake with basic information so I can narrow down whether the issue is on the parent or the children?

Otherwise I think that this is it, I cannot think about anything else to change.

And thank you for your support!

[ilyam8]

Make sure that on the Parent “stream.conf” is:

• readable by the “netdata” user.
• located in the correct directory (/opt/netdata/etc/netdata/ if your Netdata is installed in /opt/).

[tsl]

So, all machines are Debian, and the configuration is in /etc/netdata. I can see that both children are passing the correct API from the logs on the server, which means that Netdata is ready the stream.conf correctly. The permissions on the file are 644, owner by root and it works.

On the parent, I see exactly the same pattern, but somehow, it seems that it’s not parsing the stream.conf.
I’ve manually changed the ownership from root:root to netdata:netdata and kept 644 as permissions. After restarting Netdata, there is no change.

Your answer seems to point towards the possibility that stream.conf is not being parsed for any reason, which I suspect it might cause the issue I’m observing. Is there any way in Netdata to see which configuration file has loaded?

In the meantime, I will try to swap a child for the parent - just to remove any issue specific to that machine.

[tsl]

Ok, I found the issue, it was related to file encoding on the stream.conf on the parent.

After more tests and a new installation on a completely different machine I ended up with two files that looked exactly the same:

[1]
    type = api
    enabled = yes
    allow from = *
    default postpone alarms on connect seconds = 60

However, running file on the first, non-working stream.conf I got:

stream.conf: Unicode text, UTF-8 text

while running it on the working copy I got:

stream.conf.workswell: ASCII text

So I looked at the non-working stream.conf in hex:

00000000: 5b31 5d0a 2020 2020 7479 7065 203d 2061  [1].    type = a
00000010: 7069 0a20 2020 2065 6e61 626c 6564 203d  pi.    enabled =
00000020: c2a0 7965 730a 2020 2020 616c 6c6f 7720  ..yes.    allow 
00000030: 6672 6f6d 203d 202a 0a20 2020 2064 6566  from = *.    def
00000040: 6175 6c74 2070 6f73 7470 6f6e 6520 616c  ault postpone al
00000050: 6172 6d73 206f 6e20 636f 6e6e 6563 7420  arms on connect 
00000060: 7365 636f 6e64 7320 3d20 3630 0a         seconds = 60.

Now there is a non ASCII character (c2a0) right after the “enabled =” line and before the space (which is hex 2020).

So the editor doesn’t complain about it, and it’s not visible, but as you can imagine, I’ve deleted the “space” and typed it again and it worked right away.

The only explanation is that I copied that from a web page at the very beginning and stayed there.

In any case, I thank you a lot @ilyam8, because you made me focus on the stream.conf file and after all the tests, it was clear that the issue was on the server, just I could not imagine something like that, as netdata did not complain about the file and the unicode character was not visible…

[ilyam8]

Nice that you found the problem!

---

@tsl

The only explanation is that I copied that from a web page at the very beginning and stayed there.

From what page? I don’t think we have non-ASCII characters in our docs in code samples. If we have we need to fix that.

[tsl]

I have quickly checked the key pages on Netdata site and they do not present the issue, so it must have been one of the other internet pages I looked at to find possible explanations.

In any case, I hope that this post might one day help someone else

Thank you for your help!