Netdata Community

Export to VictoriaMetrics via prometheus remote write on Rpi4

Hi,

I’m trying to configure netdata running on a Raspberry Pi 4b running RPiOS (Raspbian) to export to VictoriaMetrics running on the same instance. I’m following https://learn.netdata.cloud/docs/agent/exporting/prometheus/remote_write which says to install (Google) protobuf and snappy and then reinstall netdata.

It took a while to figure out which packages I actually needed, in the end these seemed to work.

sudo apt install libsnappy-dev libsnappy1v5 protobuf-c-compiler protobuf-compiler

Once I’d installed these packages, uninstalled and reinstalled Netdata, run sudo ./edit-config exporting.conf and added the configuration shown below, and restarted Netdata I briefly see metrics coming through to VictoriaMetrics. Unfortunately the export fails after 2 minutes, which seems to be a repeatable failure, as I can get another two minutes of metrics if I stop and start Netdata again.

[exporting:global]
    enabled = yes
    send configured labels = yes
    send automatic labels = no
    update every = 10

[prometheus_remote_write:victoriametrics]
    enabled = yes
    destination = localhost:8428
    remote write URL path = /api/v1/write
    data source = average
    prefix = netdata
    hostname = localhost
    update every = 10
    buffer on failures = 10
    timeout ms = 20000
    send names instead of ids = yes
    send charts matching = *
    send hosts matching = localhost *

At this point I think it’s a Netdata issue, as I’ve also configured VictoriaMetrics to scrape from the Prometheus node exporter which is also running on localhost and that has been happily ingesting data for 40 minutes now. What I can’t find are any more useful error message to narrow this down.

From sudo journalctl -u netdata I get the following output (with the last 5 lines repeated) but no indication of why the export starts to fail.

Oct 24 12:04:47 desktop-pi systemd[1]: Starting Real time performance monitoring...
Oct 24 12:04:47 desktop-pi systemd[1]: Started Real time performance monitoring.
Oct 24 12:04:48 desktop-pi netdata[687]: CONFIG: cannot load cloud config '/var/lib/netdata/cloud.d/cloud.conf'. Running with internal defaults.
Oct 24 12:04:48 desktop-pi netdata[687]: 2020-10-24 12:04:48: netdata INFO  : MAIN : CONFIG: cannot load cloud config '/var/lib/netdata/cloud.d/cloud.conf'. Running with internal defaults.
Oct 24 12:04:48 desktop-pi netdata[687]: SIGNAL: Not enabling reaper
Oct 24 12:04:48 desktop-pi netdata[687]: 2020-10-24 12:04:48: netdata INFO  : MAIN : SIGNAL: Not enabling reaper
Oct 24 12:04:50 desktop-pi ebpf.plugin[941]: Does not have a configuration file inside `/etc/netdata/ebpf.conf. It will try to load stock file.
Oct 24 12:04:50 desktop-pi ebpf.plugin[941]: Name resolution is disabled, collector will not parser "hostnames" list.
Oct 24 12:04:50 desktop-pi ebpf.plugin[941]: The network value of CIDR 127.0.0.1/8 was updated for 127.0.0.0 .
Oct 24 12:04:50 desktop-pi ebpf.plugin[941]: PROCFILE: Cannot open file '/etc/netdata/apps_groups.conf'
Oct 24 12:04:50 desktop-pi ebpf.plugin[941]: Cannot read process groups configuration file '/etc/netdata/apps_groups.conf'. Will try '/usr/lib/netdata/conf.d/apps_groups.conf'
Oct 24 12:05:01 desktop-pi ebpf.plugin[1632]: Does not have a configuration file inside `/etc/netdata/ebpf.conf. It will try to load stock file.
Oct 24 12:05:01 desktop-pi ebpf.plugin[1632]: Name resolution is disabled, collector will not parser "hostnames" list.

Where else should I look for error messages? I don’t know for sure this is a Netdata issue and not a VictoriaMetrics issue but I have to start somewhere and try and rule out one or the other. Is there some way I can increase the log level for Netdata?

Can you please send us the logs, as shown in the documentation?

Moreover, how did you install Netdata?

Don’t worry, we will get to the bottom of this :muscle: