Metrics from child nodes not available in "/api/v1/allmetrics?format=prometheus_all_hosts" URL

Help
1

Hello,

Problem

My setup is :

  • a “central” node, that has a stream API key enabled, and that will be polled by a prometheus instance
  • several “child” nodes, that are configured to stream to “central”

I can see in “central”'s web interface all the metrics coming in from the “child” nodes.

However, when I wget the prometheus_all_hosts URL from “central”, I only get netdata_info metrics :

$ wget http://localhost:19999//api/v1/allmetrics?format=prometheus_all_hosts | grep -v 'instance="central"'
netdata_info{instance="REDACTED-preprod-ha1",application="netdata",version="v1.42.4"} 1 1695743112903
netdata_info{instance="REDACTED-prod-smtp9",application="netdata",version="v1.40.0"} 1 1695743112906
netdata_info{instance="REDACTED-prod-smtp1",application="netdata",version="v1.40.0"} 1 1695743112909
netdata_info{instance="REDACTED-prod-smtp3",application="netdata",version="v1.40.0"} 1 1695743112912
netdata_info{instance="REDACTED-prod-smtp5",application="netdata",version="v1.40.0"} 1 1695743112914
netdata_info{instance="REDACTED-prod-smtp7",application="netdata",version="v1.40.0"} 1 1695743112917
netdata_info{instance="REDACTED-prod-smtp6",application="netdata",version="v1.40.0"} 1 1695743112919
netdata_info{instance="REDACTED-prod-smtp10",application="netdata",version="v1.40.0"} 1 1695743112921
netdata_info{instance="REDACTED-prod-smtp15",application="netdata",version="v1.40.0"} 1 1695743112923
netdata_info{instance="REDACTED-prod-smtp13",application="netdata",version="v1.40.0"} 1 1695743112925
netdata_info{instance="REDACTED-prod-smtp12",application="netdata",version="v1.40.0"} 1 1695743112928
netdata_info{instance="REDACTED-prod-smtp14",application="netdata",version="v1.40.0"} 1 1695743112930
netdata_info{instance="REDACTED-prod-smtp4",application="netdata",version="v1.40.0"} 1 1695743112932
netdata_info{instance="REDACTED-prod-ha1",application="netdata",version="v1.40.0"} 1 1695743112934
netdata_info{instance="REDACTED-preprod-smtp1",application="netdata",version="v1.42.4"} 1 1695743112937
netdata_info{instance="REDACTED-prod-smtp11",application="netdata",version="v1.40.0"} 1 1695743112939
netdata_info{instance="REDACTED-prod-smtp2",application="netdata",version="v1.40.0"} 1 1695743112941
netdata_info{instance="REDACTED-preprod-smtp2",application="netdata",version="v1.42.4"} 1 1695743112943
netdata_info{instance="REDACTED-prod-smtp16",application="netdata",version="v1.40.0"} 1 1695743112946
netdata_info{instance="REDACTED-prod-smtp8",application="netdata",version="v1.40.0"} 1 1695743112948

Relevant docs you followed/actions you took to solve the issue

I tinkered around, following those links :

Here are some of my relevant configuration files (comments removed) :

  • /etc/netdata/netdata.conf
[global]
    run as user = netdata
    page cache size = 32
    dbengine multihost disk space = 256
    hostname = central
[web]
    bind to = 0.0.0.0
  • /etc/netdata/exporting.conf
[exporting:global]
    enabled = yes
  • /etc/netdata/stream.conf
[stream]
    enabled = no
    destination =
    api key =
    timeout seconds = 60
    default port = 19999
    send charts matching = *
    buffer size bytes = 10485760
    reconnect delay seconds = 5
    initial clock resync iterations = 60
[MY_API_KEY]
    type = api
    enabled = yes
    allow from = *
    health enabled by default = no
    default postpone alarms on connect seconds = 60
    seconds to replicate = 10
[MACHINE_GUID]
    type = machine
    enabled = no
    allow from = *
    postpone alarms on connect seconds = 60

What I expected to happen

I expected to see the same metrics in http://localhost:19999//api/v1/allmetrics?format=prometheus_all_hosts than in the web interface.

I should mention that I have this exact same setup elsewhere, and encounter no problem polling all the metrics with prometheus. I of course took action to match all configurations between the “central” hosts to try and fix the situation, to no avail.

Is there something I missed ?

Thank you for your help.