ben
January 26, 2023, 3:06pm
1
Problem/Question
Charts for “Firewall(netfilter) > netlink > connection tracker errors
/searches
” are gone. Before they helped me a lot.
Environment/Browser/Agent’s version etc
Linux
Google Chrome
v1.37.0-192-ge7f5028c7
What I expected to happen
Access connection tracker’s errors/searches and maybe something extra(new) would help also.
ilyam8
January 26, 2023, 3:21pm
2
Hi, @ben . Netdata gathers these statistics from /proc/net/stat/nf_conntrack . It no longer works because of this commit in Linux kernel:
Using /proc/net/nf_conntrack has been deprecated in favour of the
conntrack(8) tool.
ilyam8
January 26, 2023, 3:26pm
3
Wrong commit, it mentions /proc/net/nf_conntrack
not /proc/net/stat/nf_conntrack
but it doesn’t change the gist - no errors/searches/etc it means /proc/net/stat/nf_conntrack
is not available anymore.
ilyam8
January 26, 2023, 4:00pm
4
Also, there is nfacct collector which collects the same metrics querying netlink. See Prerequisites if you build from source.
ben
January 27, 2023, 12:30pm
5
@ilyam8 thank you for replying and helping to understand. I’ve used simple netdata installer with automatic updates enabled(Install Netdata with kickstart.sh | Learn Netdata )
/opt/netdata/usr/libexec/netdata/plugins.d # ls | grep “nfacc”
And output is empty
Do you know how should I get this plugin with simple installation without building from source?
ilyam8
January 27, 2023, 12:35pm
6
What is your install method?
/opt/netdata/usr/sbin/netdata -W buildinfo | grep Install
and can you
ls -l /opt/netdata/usr/libexec/netdata/plugins.d
ben
January 27, 2023, 12:46pm
7
/opt/netdata/usr/sbin/netdata -W buildinfo | grep Install
Install type: kickstart-static
ls -l /opt/netdata/usr/libexec/netdata/plugins.d
total 94536
-rwxr-xr-x 1 netdata netdata 4873 Jan 27 01:22 acl.sh
-rwxr-xr-x 1 netdata netdata 154 Jan 27 01:22 alarm-email.sh
-rwxr-xr-x 1 netdata netdata 139978 Jan 27 01:22 alarm-notify.sh
-rwxr-xr-x 1 netdata netdata 2155 Jan 27 01:22 alarm.sh
-rwxr-xr-x 1 netdata netdata 301 Jan 27 01:22 alarm-test.sh
-rwxr-xr-x 1 netdata netdata 7980 Jan 27 01:22 anonymous-statistics.sh
-rwsr-x--- 1 root netdata 6572040 Jan 27 01:22 apps.plugin
-rwxr-xr-x 1 netdata netdata 21010 Jan 27 01:22 cgroup-name.sh
-rwsr-x--- 1 root netdata 6467864 Jan 27 01:22 cgroup-network
-rwxr-x--- 1 netdata netdata 8768 Jan 27 01:22 cgroup-network-helper.sh
-rwxr-xr-x 1 netdata netdata 1259 Jan 27 01:22 charts.d.dryrun-helper.sh
-rwxr-x--- 1 netdata netdata 20126 Jan 27 01:22 charts.d.plugin
drwxrwxr-x 2 netdata netdata 20480 Jan 27 06:53 ebpf.d
-rwsr-x--- 1 root netdata 7309440 Jan 27 01:22 ebpf.plugin
-rwxr-x--- 1 netdata netdata 5677 Dec 9 01:25 fping.plugin
-rwxr-xr-x 1 netdata netdata 2139 Jan 27 01:22 get-kubernetes-labels.sh
-rwxr-x--- 1 netdata netdata 63053824 Jan 20 15:54 go.d.plugin
-rwxr-xr-x 1 netdata netdata 7330 Jan 27 01:22 health-cmdapi-test.sh
-rwsr-x--- 1 root netdata 77248 Jan 27 01:18 ioping
-rwxr-x--- 1 netdata netdata 5416 Jan 27 01:22 ioping.plugin
-rw-r--r-- 1 netdata netdata 6860 Jan 27 01:22 loopsleepms.sh.inc
-rwxr-x--- 1 netdata netdata 11084 Feb 19 2022 node.d.plugin
-rwsr-x--- 1 root netdata 6467272 Jan 27 01:22 perf.plugin
-rwxr-x--- 1 netdata netdata 27189 Jan 27 01:22 python.d.plugin
-rwxr-xr-x 1 netdata netdata 11020 Jan 27 01:22 request.sh
-rwsr-x--- 1 root netdata 6460960 Jan 27 01:22 slabinfo.plugin
-rwxr-xr-x 1 netdata netdata 22069 Jan 27 01:22 system-info.sh
-rwxr-xr-x 1 netdata netdata 7358 Jan 27 01:22 tc-qos-helper.sh
-rwxr-xr-x 1 netdata netdata 2915 Jan 27 01:22 template_dim.sh
ilyam8
January 27, 2023, 12:50pm
8
It looks like we don’t build nfacct for static builds. I don’t know if that is a bug or some technical limitation. cc @Austin_Hemmelgarn
ben
January 27, 2023, 2:33pm
9
Is there any other plugin that can query this information? Or only nfacct?
ilyam8
January 27, 2023, 4:48pm
10
No, there is no other plugin that can query this information.
ben
January 27, 2023, 5:08pm
11
Ok let’s wait for @Austin_Hemmelgarn then. Thank you @ilyam8
It’s less a matter of a technical limitation and more the fact that there has been relatively little demand for the nfacct plugin and it’s not trivial to get it working in the static builds, so it’s just not been a particularly high priority. I would honestly like to see it supported there myself though, and it is on the TODO list (just not with an associated issue on GitHub at the moment).
ben
January 28, 2023, 7:08am
13
Understood. Any e.t.a on this? I’m wondering whether I should bother compiling netdata for 11 nodes.
Unfortunately not at the moment. Part of the issue here is that Alpine Linux (which we use for building our static builds) does not provide static libraries for libnetfilter_acct, which means we need special handling on our end. I’ve opened an issue to track this on GitHub, which you can follow at [Feat]: nfacct.plugin support in static builds. · Issue #14367 · netdata/netdata · GitHub .
ben
February 9, 2023, 5:54pm
15
Black magic but I think I have nfacct on 2 of my nodes with netdata version v1.38.0-42-nightly
P.S. I’ve used simple one-liner bash installation
MOre information on latest comment:
cat /var/log/netdata/collector.log | grep "softirq"
Completely empty.
grep -w proc /var/log/netdata/collector.log
Returns
2023-02-09 17:39:31: ERROR : MAIN : Cannot open /proc/1325956/status
2023-02-09 17:40:25: ERROR : MAIN : PROCFILE: Cannot open file '/proc/1327355/status' (errno 2, No such file or directory)
2023-02-09 17:40:25: ERROR : MAIN : Cannot open /proc/1327355/status
2023-02-09 17:41:20: ERROR : MAIN : PROCFILE: Cannot open file '/proc/1330880/status' (errno 2, No such file …