ebpf network connection chart is not what I'm expecting

Hi all,

I have a requirement to gather the network bandwidth between an Oracle database and two specific clients. After working through this thread I’ve got the ebpf plugin working. However I can’t seem to get the chart to output what I think it should. I’m expecting the ebpf.outbound_bytes and ebpf.inbound_bytes to only show me the two hosts that I’ve specified in the ebpf.d/network.conf file to be visible. Instead it’s showing me other traffic (addresses on local interfaces, some other remote addresses, etc.)

Are my expectations incorrect? It’s entirely possible that I’m misunderstanding how things need to be configured or what the network connection part of the ebpf plugin does.

Here are some details:

netdata -W buildinfo
Version: netdata v1.36.0-326-nightly
Configure options:  '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--program-prefix=' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--datadir=/usr/share' '--includedir=/usr/include' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--prefix=/usr' '--sysconfdir=/etc' '--localstatedir=/var' '--libexecdir=/usr/libexec' '--libdir=/usr/lib' '--with-zlib' '--with-math' '--with-user=netdata' '--disable-dependency-tracking' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'CFLAGS=-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection' 'LDFLAGS=-Wl,-z,relro  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld' 'CXXFLAGS=-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection' 'PKG_CONFIG_PATH=:/usr/lib/pkgconfig:/usr/share/pkgconfig'
Install type: binpkg-rpm
    Binary architecture: x86_64
    Packaging distro:
Features:
    dbengine:                   YES
    Native HTTPS:               YES
    Netdata Cloud:              YES
    ACLK:                       YES
    TLS Host Verification:      YES
    Machine Learning:           YES
    Stream Compression:         NO
Libraries:
    protobuf:                YES (system)
    jemalloc:                NO
    JSON-C:                  YES
    libcap:                  NO
    libcrypto:               YES
    libm:                    YES
    tcalloc:                 NO
    zlib:                    YES
Plugins:
    apps:                    YES
    cgroup Network Tracking: YES
    CUPS:                    YES
    EBPF:                    YES
    IPMI:                    YES
    NFACCT:                  NO
    perf:                    YES
    slabinfo:                YES
    Xen:                     NO
    Xen VBD Error Tracking:  NO
Exporters:
    AWS Kinesis:             NO
    GCP PubSub:              NO
    MongoDB:                 NO
    Prometheus Remote Write: YES
Debug/Developer Features:
    Trace Allocations:       NO
# uname -a
Linux viewdb04.prod 5.4.17-2102.204.4.4.el8uek.x86_64 #2 SMP Tue Aug 17 20:25:27 PDT 2021 x86_64 x86_64 x86_64 GNU/Linux
# cat ebpf.d/network.conf  | grep -v '^#'
[global]
    bandwidth table size = 16384
    ipv4 connection table size = 16384
    ipv6 connection table size = 16384
    udp connection table size = 4096
    ebpf type format = auto
    ebpf co-re tracing = trampoline

[network connections]
    maximum dimensions = 50
    resolve hostnames = no
    resolve service names = no
    #ports = *
    #ips = !127.0.0.1/8 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 fc00::/7 !::1/128
    #hostnames = *
    ips = 10.20.6.23 10.20.6.24

[service name]
    19999 = Netdata

Those addresses are making connections:

# ss -tapn | grep '10.20.6.23\|10.20.6.24' | wc -l
292

Screenshots of what I’m seeing.


Hello @mjtice ,

I am glad you can have eBPF.plugin running on Oracle Linux.
Now about the connections, that are not enabling by default, this is part of a project that we are still building. We want to deliver for our users a complete Network Viewer. Considering the charts and configuration I am seeing it is possible that you reach an area of the code that was still not completely developed and thanks this you found this bug.

I will take a look and bring an update.

Best regards!

thanks again, @Thiago_Marques_0

1 Like

Hello @mjtice ,

The fix for the issue you reported need three steps:

1 - Work with legacy codes, this is almost ready with this PR Fix Network Viewer by thiagoftsm · Pull Request #317 · netdata/kernel-collector · GitHub, I am testing now to verify that I did not miss nothing. This PR already brings part of the fix for the reported issue. And improvement for socket monitoring.
2 - When this PR is merged, I will work with CO-RE codes and kernels newer than 5.5 that has BTF codes.
3 - Finally we will have a PR in netdata/netdata bringing changes for two previous repositories and a small adjust in the code to match what you need.

I will keep the issue updated when next steps are finalized.

Best regards!

Perfect! Thanks, @Thiago_Marques_0

1 Like

Hello @mjtice ,

I merged the second PR now Fix nv by thiagoftsm · Pull Request #31 · netdata/ebpf-co-re · GitHub. As soon this another PR Improve eBPF exit by thiagoftsm · Pull Request #14012 · netdata/netdata · GitHub is merged, I will bring the final PR to fix this issue. :handshake:

Best regards!

Hello @mjtice ,

As I described in the latest PR eBPF (memory an NV) by thiagoftsm · Pull Request #14131 · netdata/netdata · GitHub, I am only waiting for a PR to be merged and I will finish it.

Best regards!

Hello @mjtice ,

Happy new year!

Yesterday we merged the PR that fixes the issue you reported and others.
I hope to bring more updates to improve connection monitoring during this year.

Best regards!

Great, thanks @Thiago_Marques_0 ! I’ll give it a spin as soon as I get a chance.

1 Like