Netfilter → bandwidth accounting chart missing, but plugin debug mode producing CHART & SET commands (solved)

Help
1

Solution

Add CapabilityBoundingSet=CAP_NET_ADMIN to netdata.service. Not sure if this is the minimal additional capability to read nfacct data under Ubuntu 16.04.

(Naturally, found the solution while typing up this question, so posting here to leave an answer for the next person to encounter this).

Environment

  • Netdata 1.29.3 installed from Packagecloud repository.
  • Ubuntu 16.04 (kernel 4.4.0-197)
  • is a VM, hypervisor is QEMU/KVM with libvirt on Debian 10 (Buster).
  • this is a Chef test kitchen

Problem/Question

On other machines (running Debian 10, instead of Ubuntu 16.04), setting up nfacct objects + iptables rules to increment their counters was enough to get Netdata to produce a bandwidth accounting chart under Firewall (netfilter). This is labeled as coming from the nfacct plugin.

On this VM, I get the connection tracker and netlink charts, but not the bandwidth accounting chart. Just to be sure it wasn’t a browser issue I:

  • tried shift-refresh, cleared cache cache, & tried a different browner
  • used curl to pull all metrics from the API, to see if NETDATA_NETFILTER_NFACCT_BYTES_* entries were present, and are not (but they are on the working machine): curl "http://192.168.121.137:19999/api/v1/allmetrics?format=shell&variables=no&help=no&types=no&timestamps=yes&names=yes&oldunits=yes&hideunits=yes&data=average" -H "accept: */*" | grep NFACCT_BYTES

To make sure it’s not a plugin issue, I:

  • ran /usr/libexec/netdata/plugins.d/nfacct.plugin 1 debug. The output shows CHART netfilter.nfacct_bytes… and DIMENSION commands at start, and BEGIN/SET/END commands every second with updated byte & packet counters.
  • looked in /var/log/netdata/error.log for anything about nfacct, netfilter, or bandwidth. There are some messages about it restarting, and about charts already existing. (See below)
  • pointed strace at the running nfacct plugin and… oh… it isn’t producing bandwidth messages. Errr, looks like we’ve found the problem.

Log messages

root@antz-us-cl-testkitchen:~# grep -Ei 'nfacct|netfilter|bandwidth' /var/log/netdata/error.log 
2021-03-08 08:59:02: netdata ERROR : PLUGINSD[nfacct] : read failed: end of file (errno 9, Bad file descriptor)
2021-03-08 08:59:02: netdata INFO  : PLUGINSD[nfacct] : PARSER ended
2021-03-08 08:59:02: netdata ERROR : PLUGINSD[nfacct] : '/usr/libexec/netdata/plugins.d/nfacct.plugin' (pid 16596) disconnected after 72005 successful data collections (ENDs).
2021-03-08 08:59:03: netdata INFO  : PLUGINSD[nfacct] : connected to '/usr/libexec/netdata/plugins.d/nfacct.plugin' running on pid 19827
2021-03-08 08:59:04: netdata INFO  : PLUGINSD[nfacct] : RRDSET: chart name 'netfilter.netlink_new' on host 'antz-us-cl-testkitchen' already exists.
2021-03-08 08:59:04: netdata INFO  : PLUGINSD[nfacct] : RRDSET: chart name 'netfilter.netlink_changes' on host 'antz-us-cl-testkitchen' already exists.
2021-03-08 08:59:04: netdata INFO  : PLUGINSD[nfacct] : RRDSET: chart name 'netfilter.netlink_search' on host 'antz-us-cl-testkitchen' already exists.
2021-03-08 08:59:04: netdata INFO  : PLUGINSD[nfacct] : RRDSET: chart name 'netfilter.netlink_errors' on host 'antz-us-cl-testkitchen' already exists.
2021-03-08 08:59:04: netdata INFO  : PLUGINSD[nfacct] : RRDSET: chart name 'netfilter.netlink_expect' on host 'antz-us-cl-testkitchen' already exists.
2021-03-08 12:59:04: nfacct.plugin INFO  : MAIN : NFACCT process exiting
2021-03-08 12:59:04: netdata ERROR : PLUGINSD[nfacct] : read failed: end of file (errno 9, Bad file descriptor)
2021-03-08 12:59:04: netdata INFO  : PLUGINSD[nfacct] : PARSER ended
2021-03-08 12:59:04: netdata ERROR : PLUGINSD[nfacct] : '/usr/libexec/netdata/plugins.d/nfacct.plugin' (pid 19827) disconnected after 72005 successful data collections (ENDs).
2021-03-08 12:59:05: netdata INFO  : PLUGINSD[nfacct] : connected to '/usr/libexec/netdata/plugins.d/nfacct.plugin' running on pid 22948
2021-03-08 12:59:06: netdata INFO  : PLUGINSD[nfacct] : RRDSET: chart name 'netfilter.netlink_new' on host 'antz-us-cl-testkitchen' already exists.
2021-03-08 12:59:06: netdata INFO  : PLUGINSD[nfacct] : RRDSET: chart name 'netfilter.netlink_changes' on host 'antz-us-cl-testkitchen' already exists.
2021-03-08 12:59:06: netdata INFO  : PLUGINSD[nfacct] : RRDSET: chart name 'netfilter.netlink_search' on host 'antz-us-cl-testkitchen' already exists.
2021-03-08 12:59:06: netdata INFO  : PLUGINSD[nfacct] : RRDSET: chart name 'netfilter.netlink_errors' on host 'antz-us-cl-testkitchen' already exists.
2021-03-08 12:59:06: netdata INFO  : PLUGINSD[nfacct] : RRDSET: chart name 'netfilter.netlink_expect' on host 'antz-us-cl-testkitchen' already exists.
2021-03-08 15:25:09: netdata ERROR : PLUGINSD[nfacct] : read failed: end of file (errno 9, Bad file descriptor)
2021-03-08 15:25:09: netdata INFO  : PLUGINSD[nfacct] : PARSER ended
2021-03-08 15:25:09: netdata ERROR : PLUGINSD[nfacct] : '/usr/libexec/netdata/plugins.d/nfacct.plugin' (pid 22948) disconnected after 43815 successful data collections (ENDs).
2021-03-08 15:25:09: netdata INFO  : PLUGINSD : stopping plugin thread: plugin:nfacct
2021-03-08 15:25:09: netdata INFO  : PLUGINSD[nfacct] : data collection thread exiting
2021-03-08 15:25:09: netdata INFO  : PLUGINSD[nfacct] : killing child process pid 22948
2021-03-08 15:25:09: netdata INFO  : PLUGINSD[nfacct] : thread with task id 1415 finished
2021-03-08 15:25:23: netdata INFO  : PLUGINSD[nfacct] : thread created with task id 25160
2021-03-08 15:25:23: netdata INFO  : PLUGINSD[nfacct] : set name of thread 25160 to PLUGINSD[nfacct
2021-03-08 15:25:23: netdata INFO  : PLUGINSD[nfacct] : connected to '/usr/libexec/netdata/plugins.d/nfacct.plugin' running on pid 25178
1 Like
2

Hi @derobert_work. That is very nice that you’ve figured it out! Are you willing to make a PR? (update this file)

3

Sure, beats carrying the change locally. Possible I won’t get to it until Wednesday though.

1 Like
4

That’s awesome, thanks!

If you need any help along the way, just ping us!

5

FYI, this is not fixed. Just installed with kickstart and I get nfacct values only if I update the netdata.service as instructed here. Ubuntu 20.04.

6

Fixed in fix(nfacct.plugin): Netfilter accounting data collection by ilyam8 · Pull Request #11951 · netdata/netdata · GitHub