Hi,
I’m trying to configure netdata running on a Raspberry Pi 4b running RPiOS (Raspbian) to export to VictoriaMetrics running on the same instance. I’m following Export metrics to Prometheus remote write providers | Learn Netdata which says to install (Google) protobuf and snappy and then reinstall netdata.
It took a while to figure out which packages I actually needed, in the end these seemed to work.
sudo apt install libsnappy-dev libsnappy1v5 protobuf-c-compiler protobuf-compiler
Once I’d installed these packages, uninstalled and reinstalled Netdata, run sudo ./edit-config exporting.conf and added the configuration shown below, and restarted Netdata I briefly see metrics coming through to VictoriaMetrics. Unfortunately the export fails after 2 minutes, which seems to be a repeatable failure, as I can get another two minutes of metrics if I stop and start Netdata again.
[exporting:global]
enabled = yes
send configured labels = yes
send automatic labels = no
update every = 10
[prometheus_remote_write:victoriametrics]
enabled = yes
destination = localhost:8428
remote write URL path = /api/v1/write
data source = average
prefix = netdata
hostname = localhost
update every = 10
buffer on failures = 10
timeout ms = 20000
send names instead of ids = yes
send charts matching = *
send hosts matching = localhost *
At this point I think it’s a Netdata issue, as I’ve also configured VictoriaMetrics to scrape from the Prometheus node exporter which is also running on localhost and that has been happily ingesting data for 40 minutes now. What I can’t find are any more useful error message to narrow this down.
From sudo journalctl -u netdata I get the following output (with the last 5 lines repeated) but no indication of why the export starts to fail.
Oct 24 12:04:47 desktop-pi systemd[1]: Starting Real time performance monitoring...
Oct 24 12:04:47 desktop-pi systemd[1]: Started Real time performance monitoring.
Oct 24 12:04:48 desktop-pi netdata[687]: CONFIG: cannot load cloud config '/var/lib/netdata/cloud.d/cloud.conf'. Running with internal defaults.
Oct 24 12:04:48 desktop-pi netdata[687]: 2020-10-24 12:04:48: netdata INFO : MAIN : CONFIG: cannot load cloud config '/var/lib/netdata/cloud.d/cloud.conf'. Running with internal defaults.
Oct 24 12:04:48 desktop-pi netdata[687]: SIGNAL: Not enabling reaper
Oct 24 12:04:48 desktop-pi netdata[687]: 2020-10-24 12:04:48: netdata INFO : MAIN : SIGNAL: Not enabling reaper
Oct 24 12:04:50 desktop-pi ebpf.plugin[941]: Does not have a configuration file inside `/etc/netdata/ebpf.conf. It will try to load stock file.
Oct 24 12:04:50 desktop-pi ebpf.plugin[941]: Name resolution is disabled, collector will not parser "hostnames" list.
Oct 24 12:04:50 desktop-pi ebpf.plugin[941]: The network value of CIDR 127.0.0.1/8 was updated for 127.0.0.0 .
Oct 24 12:04:50 desktop-pi ebpf.plugin[941]: PROCFILE: Cannot open file '/etc/netdata/apps_groups.conf'
Oct 24 12:04:50 desktop-pi ebpf.plugin[941]: Cannot read process groups configuration file '/etc/netdata/apps_groups.conf'. Will try '/usr/lib/netdata/conf.d/apps_groups.conf'
Oct 24 12:05:01 desktop-pi ebpf.plugin[1632]: Does not have a configuration file inside `/etc/netdata/ebpf.conf. It will try to load stock file.
Oct 24 12:05:01 desktop-pi ebpf.plugin[1632]: Name resolution is disabled, collector will not parser "hostnames" list.
Where else should I look for error messages? I don’t know for sure this is a Netdata issue and not a VictoriaMetrics issue but I have to start somewhere and try and rule out one or the other. Is there some way I can increase the log level for Netdata?
