EBPF Errors

Hi Everyone,

I have a scenario where I’ve been asked to determine how much traffic a particular subset of machines send/receive on specific interfaces ONLY between certain other machines. Looking through the EBPF documentation it seems like enabling network connections is the way to go.

However, after enabling the ebpf plugin and restarting the netdata service it looks like that particular portion of the plugin (and others) fail. Here are the errors from both the error.log and from running the command manually.

$ su - netdata
$ export NETDATA_STOCK_CONFIG_DIR=/etc/netdata/conf.d/
$ /usr/libexec/netdata/plugins.d/ebpf.plugin
2022-10-28 10:38:35:  INFO  : MAIN : Does not have a configuration file inside `/etc/netdata/ebpf.d.conf. It will try to load stock file.
2022-10-28 10:38:35:  INFO  : MAIN : Name resolution is disabled, collector will not parser "hostnames" list.
2022-10-28 10:38:35:  INFO  : MAIN : The network value of CIDR 127.0.0.1/8 was updated for 127.0.0.0 .
2022-10-28 10:38:35:  ERROR : MAIN : Cannot load the ebpf configuration file process.conf (errno 2, No such file or directory)
2022-10-28 10:38:35:  ERROR : MAIN : Cannot load the ebpf configuration file cachestat.conf (errno 2, No such file or directory)
2022-10-28 10:38:35:  ERROR : MAIN : Cannot load the ebpf configuration file sync.conf (errno 2, No such file or directory)
2022-10-28 10:38:35:  ERROR : MAIN : Cannot load the ebpf configuration file dcstat.conf (errno 2, No such file or directory)
2022-10-28 10:38:35:  ERROR : MAIN : Cannot load the ebpf configuration file swap.conf (errno 2, No such file or directory)
2022-10-28 10:38:35:  ERROR : MAIN : Cannot load the ebpf configuration file vfs.conf (errno 2, No such file or directory)
2022-10-28 10:38:35:  ERROR : MAIN : Cannot load the ebpf configuration file filesystem.conf (errno 2, No such file or directory)
2022-10-28 10:38:35:  ERROR : MAIN : Cannot load the ebpf configuration file disk.conf (errno 2, No such file or directory)
2022-10-28 10:38:35:  ERROR : MAIN : Cannot load the ebpf configuration file mount.conf (errno 2, No such file or directory)
2022-10-28 10:38:35:  ERROR : MAIN : Cannot load the ebpf configuration file fd.conf (errno 2, No such file or directory)
2022-10-28 10:38:35:  ERROR : MAIN : Cannot load the ebpf configuration file hardirq.conf (errno 2, No such file or directory)
2022-10-28 10:38:35:  ERROR : MAIN : Cannot load the ebpf configuration file softirq.conf (errno 2, No such file or directory)
2022-10-28 10:38:35:  ERROR : MAIN : Cannot load the ebpf configuration file oomkill.conf (errno 2, No such file or directory)
2022-10-28 10:38:35:  ERROR : MAIN : Cannot load the ebpf configuration file shm.conf (errno 2, No such file or directory)
2022-10-28 10:38:35:  ERROR : MAIN : Cannot load the ebpf configuration file mdflush.conf (errno 2, No such file or directory)
2022-10-28 10:38:35:  INFO  : MAIN : Cannot read process groups configuration file '/etc/netdata/apps_groups.conf'. Will try '/etc/netdata/conf.d//apps_groups.conf'
2022-10-28 10:38:35: ebpf.plugin INFO  : EBPF PROCESS : thread created with task id 2645227
2022-10-28 10:38:35: ebpf.plugin INFO  : EBPF SOCKET : thread created with task id 2645228
2022-10-28 10:38:35: ebpf.plugin INFO  : EBPF PROCESS : set name of thread 2645227 to EBPF PROCESS
2022-10-28 10:38:35: ebpf.plugin INFO  : EBPF CACHESTAT : thread created with task id 2645229
2022-10-28 10:38:35: ebpf.plugin INFO  : EBPF SYNC : thread created with task id 2645230
2022-10-28 10:38:35: ebpf.plugin INFO  : EBPF SOCKET : set name of thread 2645228 to EBPF SOCKET
2022-10-28 10:38:35: ebpf.plugin INFO  : EBPF VFS : thread created with task id 2645233
2022-10-28 10:38:35: ebpf.plugin INFO  : EBPF CACHESTAT : set name of thread 2645229 to EBPF CACHESTAT
2022-10-28 10:38:35: ebpf.plugin INFO  : EBPF SWAP : thread created with task id 2645232
2022-10-28 10:38:35: ebpf.plugin INFO  : EBPF SYNC : set name of thread 2645230 to EBPF SYNC
2022-10-28 10:38:35: ebpf.plugin INFO  : EBPF DCSTAT : thread created with task id 2645231
2022-10-28 10:38:35: ebpf.plugin INFO  : EBPF SOCKET : Name resolution is disabled, collector will not parser "hostnames" list.
2022-10-28 10:38:35: ebpf.plugin INFO  : EBPF FILESYSTEM : thread created with task id 2645234
2022-10-28 10:38:35: ebpf.plugin INFO  : EBPF SOFTIRQ : thread created with task id 2645239
2022-10-28 10:38:35: ebpf.plugin INFO  : EBPF DISK : thread created with task id 2645235
2022-10-28 10:38:35: ebpf.plugin INFO  : EBPF SWAP : set name of thread 2645232 to EBPF SWAP
2022-10-28 10:38:35: ebpf.plugin INFO  : EBPF MOUNT : thread created with task id 2645236
2022-10-28 10:38:35: ebpf.plugin INFO  : EBPF FD : thread created with task id 2645237
2022-10-28 10:38:35: ebpf.plugin INFO  : EBPF DCSTAT : set name of thread 2645231 to EBPF DCSTAT
2022-10-28 10:38:35: ebpf.plugin INFO  : EBPF HARDIRQ : thread created with task id 2645238
2022-10-28 10:38:35: ebpf.plugin INFO  : EBPF SOCKET : The network value of CIDR 127.0.0.1/8 was updated for 127.0.0.0 .
2022-10-28 10:38:35: ebpf.plugin INFO  : EBPF SOCKET : The range/value (127.0.0.1/8) is inside the range/value (127.0.0.1/8) already inserted, it will be ignored.
2022-10-28 10:38:35: ebpf.plugin INFO  : EBPF SOCKET : The range/value (10.0.0.0/8) is inside the range/value (10.0.0.0/8) already inserted, it will be ignored.
2022-10-28 10:38:35: ebpf.plugin INFO  : EBPF SOCKET : The range/value (172.16.0.0/12) is inside the range/value (172.16.0.0/12) already inserted, it will be ignored.
2022-10-28 10:38:35: ebpf.plugin INFO  : EBPF SOCKET : The range/value (192.168.0.0/16) is inside the range/value (192.168.0.0/16) already inserted, it will be ignored.
2022-10-28 10:38:35: ebpf.plugin INFO  : EBPF SOCKET : The range/value (fc00::/7) is inside the range/value (fc00::/7) already inserted, it will be ignored.
2022-10-28 10:38:35: ebpf.plugin INFO  : EBPF SOCKET : The range/value (::1/128) is inside the range/value (::1/128) already inserted, it will be ignored.
2022-10-28 10:38:35: ebpf.plugin INFO  : EBPF MDFLUSH : thread created with task id 2645242
2022-10-28 10:38:35: ebpf.plugin INFO  : EBPF MOUNT : set name of thread 2645236 to EBPF MOUNT
2022-10-28 10:38:35: ebpf.plugin INFO  : EBPF FD : set name of thread 2645237 to EBPF FD
2022-10-28 10:38:35: ebpf.plugin INFO  : EBPF HARDIRQ : set name of thread 2645238 to EBPF HARDIRQ
2022-10-28 10:38:35: ebpf.plugin INFO  : EBPF FILESYSTEM : set name of thread 2645234 to EBPF FILESYSTEM
2022-10-28 10:38:35: ebpf.plugin INFO  : EBPF VFS : set name of thread 2645233 to EBPF VFS
2022-10-28 10:38:35: ebpf.plugin INFO  : EBPF OOMKILL : thread created with task id 2645240
2022-10-28 10:38:35: ebpf.plugin INFO  : EBPF SOFTIRQ : set name of thread 2645239 to EBPF SOFTIRQ
2022-10-28 10:38:35: ebpf.plugin INFO  : EBPF SHM : thread created with task id 2645241
2022-10-28 10:38:35: ebpf.plugin INFO  : EBPF DISK : set name of thread 2645235 to EBPF DISK
2022-10-28 10:38:35: ebpf.plugin INFO  : EBPF MDFLUSH : set name of thread 2645242 to EBPF MDFLUSH
2022-10-28 10:38:35: ebpf.plugin INFO  : EBPF OOMKILL : set name of thread 2645240 to EBPF OOMKILL
2022-10-28 10:38:35: ebpf.plugin INFO  : EBPF OOMKILL : Disabling OOMKILL thread, because apps integration is completely disabled.
2022-10-28 10:38:35: ebpf.plugin INFO  : EBPF SHM : set name of thread 2645241 to EBPF SHM
2022-10-28 10:38:35: ebpf.plugin INFO  : SOFTIRQ KERNEL : thread created with task id 2645243
2022-10-28 10:38:35: ebpf.plugin INFO  : SOFTIRQ KERNEL : set name of thread 2645243 to SOFTIRQ KERNEL
CHART system.process_thread '' 'Start process' 'calls/s' 'processes' '' 'line' 21002 5 '' 'ebpf.plugin' 'process'
DIMENSION _do_fork process absolute 1 1
DIMENSION sys_clone thread absolute 1 1
CHART system.exit '' 'Exit process' 'calls/s' 'processes' '' 'line' 21003 5 '' 'ebpf.plugin' 'process'
DIMENSION do_exit process absolute 1 1
DIMENSION release_task task absolute 1 1
CHART system.process_status '' 'Process not closed' 'difference' 'processes' '' line 21004 5 '' 'ebpf.plugin' 'process'
DIMENSION process '' absolute 1 1
DIMENSION zombie '' absolute 1 1
CHART netdata.ebpf_threads '' 'Threads info.' 'threads' 'ebpf' '' 'line' 140000 5 '' 'ebpf.plugin' 'process'
DIMENSION total total absolute 1 1
DIMENSION running running absolute 1 1
CHART netdata.ebpf_load_methods '' 'Load info.' 'methods' 'ebpf' '' 'line' 140001 5 '' 'ebpf.plugin' 'process'
DIMENSION legacy legacy absolute 1 1
DIMENSION co-re co-re absolute 1 1
2022-10-28 10:38:35: ebpf.plugin INFO  : EBPF OOMKILL : thread with task id 2645240 finished
CHART system.softirq_latency '' 'Software IRQ latency' 'milliseconds' 'softirqs' '' 'stacked' 951 5 '' 'ebpf.plugin' 'softirq'
DIMENSION HI HI incremental 1 1
DIMENSION TIMER TIMER incremental 1 1
DIMENSION NET_TX NET_TX incremental 1 1
DIMENSION NET_RX NET_RX incremental 1 1
DIMENSION BLOCK BLOCK incremental 1 1
DIMENSION IRQ_POLL IRQ_POLL incremental 1 1
2022-10-28 10:38:35: ebpf.plugin INFO  : EBPF CGROUP : DIMENSION TASKLET TASKLET incremental 1 1
DIMENSION SCHED SCHED incremental 1 1
thread created with task id 2645244DIMENSION HRTIMER HRTIMER incremental 1 1

DIMENSION RCU RCU incremental 1 1
2022-10-28 10:38:35: ebpf.plugin INFO  : EBPF CGROUP : set name of thread 2645244 to EBPF CGROUP
CHART filesystem.file_descriptor '' 'Open and close calls' 'calls/s' 'file_access' '' 'line' 2195 5 '' 'ebpf.plugin' 'filedescriptor'
DIMENSION do_sys_open open incremental 1 1
DIMENSION __close_fd close incremental 1 1
2022-10-28 10:38:35: ebpf.plugin INFO  : FD KERNEL : thread created with task id 2645245
2022-10-28 10:38:35: ebpf.plugin INFO  : FD KERNEL : set name of thread 2645245 to FD KERNEL
CHART system.swapcalls '' 'Calls to access swap memory' 'calls/s' 'swap' '' 'line' 202 5 '' 'ebpf.plugin' 'swap'
DIMENSION read read incremental 1 1
DIMENSION write write incremental 1 1
2022-10-28 10:38:35: ebpf.plugin INFO  : SWAP KERNEL : thread created with task id 2645246
2022-10-28 10:38:35: ebpf.plugin INFO  : SWAP KERNEL : set name of thread 2645246 to SWAP KERNEL
CHART mount_points.call '' 'Calls to mount and umount syscalls' 'calls/s' 'mount (eBPF)' '' 'line' 2190 5 '' 'ebpf.plugin' 'mount'
DIMENSION mount mount incremental 1 1
DIMENSION umount umount incremental 1 1
CHART mount_points.error '' 'Errors to mount and umount file systems' 'calls/s' 'mount (eBPF)' '' 'line' 2191 5 '' 'ebpf.plugin' 'mount'
DIMENSION mount mount incremental 1 1
DIMENSION umount umount incremental 1 1
2022-10-28 10:38:35: ebpf.plugin INFO  : MOUNT KERNEL : thread created with task id 2645247
2022-10-28 10:38:35: ebpf.plugin INFO  : MOUNT KERNEL : set name of thread 2645247 to MOUNT KERNEL
2022-10-28 10:38:35: ebpf.plugin INFO  : DISK KERNEL : thread created with task id 2645248
2022-10-28 10:38:35: ebpf.plugin INFO  : DISK KERNEL : set name of thread 2645248 to DISK KERNEL
CHART mem.cachestat_ratio '' 'Hit ratio' '%' 'page_cache' '' 'line' 21100 5 '' 'ebpf.plugin' 'cachestat'
DIMENSION ratio ratio absolute 1 1
CHART mem.cachestat_dirties '' 'Number of dirty pages' 'pages/s' 'page_cache' '' 'line' 21101 5 '' 'ebpf.plugin' 'cachestat'
DIMENSION dirty dirty incremental 1 1
CHART mem.cachestat_hits '' 'Number of accessed files' 'hits/s' 'page_cache' '' 'line' 21102 5 '' 'ebpf.plugin' 'cachestat'
DIMENSION hit hit absolute 1 1
CHART mem.cachestat_misses '' 'Files out of page cache' 'misses/s' 'page_cache' '' 'line' 21103 5 '' 'ebpf.plugin' 'cachestat'
DIMENSION miss miss absolute 1 1
2022-10-28 10:38:35: ebpf.plugin INFO  : CACHESTAT KERNEL : thread created with task id 2645249
2022-10-28 10:38:35: ebpf.plugin INFO  : CACHESTAT KERNEL : set name of thread 2645249 to CACHESTAT KERNE
CHART system.shared_memory_calls '' 'Calls to shared memory system calls' 'calls/s' 'ipc shared memory' '' 'line' 1207 5 '' 'ebpf.plugin' 'shm'
DIMENSION get get incremental 1 1
DIMENSION at at incremental 1 1
DIMENSION dt dt incremental 1 1
DIMENSION ctl ctl incremental 1 1
2022-10-28 10:38:35: ebpf.plugin INFO  : SHM KERNEL : thread created with task id 2645250
2022-10-28 10:38:35: ebpf.plugin INFO  : SHM KERNEL : set name of thread 2645250 to SHM KERNEL
CHART mdstat.mdstat_flush '' 'MD flushes' 'flushes' 'flush (eBPF)' 'md.flush' 'stacked' 2101 5 '' 'ebpf.plugin' 'mdflush'
2022-10-28 10:38:35: ebpf.plugin INFO  : MDFLUSH KERNEL : thread created with task id 2645251
2022-10-28 10:38:35: ebpf.plugin INFO  : MDFLUSH KERNEL : set name of thread 2645251 to MDFLUSH KERNEL
CHART system.hardirq_latency '' 'Hardware IRQ latency' 'milliseconds' 'interrupts' '' 'stacked' 2900 5 '' 'ebpf.plugin' 'hardirq'
DIMENSION apic_thermal apic_thermal incremental 1 1
DIMENSION apic_threshold apic_threshold incremental 1 1
DIMENSION apic_error apic_error incremental 1 1
DIMENSION apic_deferred_error apic_deferred_error incremental 1 1
2022-10-28 10:38:35: ebpf.plugin INFO  : HARDIRQ KERNEL : DIMENSION apic_spurious apic_spurious incremental 1 1
DIMENSION func_call func_call incremental 1 1
thread created with task id 2645252DIMENSION func_call_single func_call_single incremental 1 1

DIMENSION reschedule reschedule incremental 1 1
DIMENSION local_timer local_timer incremental 1 1
DIMENSION irq_work irq_work incremental 1 1
2022-10-28 10:38:35: ebpf.plugin INFO  : HARDIRQ KERNEL : DIMENSION x86_platform_ipi x86_platform_ipi incremental 1 1
set name of thread 2645252 to HARDIRQ KERNEL
CHART mem.file_sync '' 'Monitor calls for <code>fsync(2)</code> and <code>fdatasync(2)</code>.' 'calls/s' 'synchronization (eBPF)' '' 'line' 21300 5 '' 'ebpf.plugin' 'sync'
DIMENSION fsync fsync incremental 1 1
DIMENSION fdatasync fdatasync incremental 1 1
CHART mem.memory_map '' 'Monitor calls for <code>msync(2)</code>.' 'calls/s' 'synchronization (eBPF)' '' 'line' 21301 5 '' 'ebpf.plugin' 'sync'
DIMENSION msync msync incremental 1 1
CHART mem.sync '' 'Monitor calls for <code>sync(2)</code> and <code>syncfs(2)</code>.' 'calls/s' 'synchronization (eBPF)' '' 'line' 21302 5 '' 'ebpf.plugin' 'sync'
DIMENSION sync sync incremental 1 1
DIMENSION syncfs syncfs incremental 1 1
CHART mem.file_segment '' 'Monitor calls for <code>sync_file_range(2)</code>.' 'calls/s' 'synchronization (eBPF)' '' 'line' 21303 5 '' 'ebpf.plugin' 'sync'
DIMENSION sync_file_range sync_file_range incremental 1 1
2022-10-28 10:38:35: ebpf.plugin INFO  : SYNC KERNEL : thread created with task id 2645253
2022-10-28 10:38:35: ebpf.plugin INFO  : SYNC KERNEL : set name of thread 2645253 to SYNC KERNEL
CHART filesystem.dc_hit_ratio '' 'Percentage of files inside directory cache' '%' 'directory cache (eBPF)' '' 'line' 21200 5 '' 'ebpf.plugin' 'dcstat'
DIMENSION ratio ratio absolute 1 1
CHART filesystem.dc_reference '' 'Variables used to calculate hit ratio.' 'files' 'directory cache (eBPF)' '' 'line' 21201 5 '' 'ebpf.plugin' 'dcstat'
DIMENSION reference reference absolute 1 1
DIMENSION slow slow absolute 1 1
DIMENSION miss miss absolute 1 1
2022-10-28 10:38:35: ebpf.plugin INFO  : DCSTAT KERNEL : thread created with task id 2645254
2022-10-28 10:38:35: ebpf.plugin INFO  : DCSTAT KERNEL : set name of thread 2645254 to DCSTAT KERNEL
CHART filesystem.vfs_deleted_objects '' 'Remove files' 'calls/s' 'vfs' '' 'line' 2150 5 '' 'ebpf.plugin' 'vfs'
DIMENSION vfs_unlink delete incremental 1 1
CHART filesystem.vfs_io '' 'Calls to IO' 'calls/s' 'vfs' '' 'line' 2151 5 '' 'ebpf.plugin' 'vfs'
DIMENSION vfs_read read incremental 1 1
DIMENSION vfs_write write incremental 1 1
CHART filesystem.vfs_io_bytes '' 'Bytes written and read' 'bytes/s' 'vfs' '' line 2152 5 '' 'ebpf.plugin' 'filesystem'
DIMENSION vfs_read read incremental 1 1
DIMENSION vfs_write write incremental -1 1
CHART filesystem.vfs_fsync '' 'Calls for <code>vfs_fsync</code>' 'calls/s' 'vfs' '' 'line' 2154 5 '' 'ebpf.plugin' 'vfs'
DIMENSION vfs_fsync fsync incremental 1 1
CHART filesystem.vfs_open '' 'Calls for <code>vfs_open</code>' 'calls/s' 'vfs' '' 'line' 2156 5 '' 'ebpf.plugin' 'vfs'
DIMENSION vfs_open open incremental 1 1
CHART filesystem.vfs_create '' 'Calls for <code>vfs_create</code>' 'calls/s' 'vfs' '' 'line' 2158 5 '' 'ebpf.plugin' 'vfs'
DIMENSION vfs_create create incremental 1 1
2022-10-28 10:38:35: ebpf.plugin INFO  : VFS KERNEL : thread created with task id 2645261
2022-10-28 10:38:35: ebpf.plugin INFO  : VFS KERNEL : set name of thread 2645261 to VFS KERNEL
libbpf: prog 'netdata_inet_csk_accept_fentry': BPF program load failed: Invalid argument
libbpf: failed to load program 'netdata_inet_csk_accept_fentry'
libbpf: failed to load object 'socket_bpf'
libbpf: failed to load BPF skeleton 'socket_bpf': -22
failed to load BPF object: -22
2022-10-28 10:38:35: ebpf.plugin ERROR : EBPF SOCKET : Cannot open or load BPF file for thread socket (errno 22, Invalid argument)
2022-10-28 10:38:35: ebpf.plugin INFO  : EBPF SOCKET : thread with task id 2645228 finished
libbpf: kprobe perf_event_open() failed: No such file or directory
libbpf: prog 'netdata_nfs4_file_open': failed to create kprobe 'nfs4_file_open+0x0' perf event: No such file or directory
libbpf: kprobe perf_event_open() failed: No such file or directory
libbpf: prog 'netdata_ret_nfs4_file_open': failed to create kretprobe 'nfs4_file_open+0x0' perf event: No such file or directory

[ snip the chart output ]

Details are below:

# netdata -W buildinfo
Version: netdata v1.36.1
Configure options:  '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--program-prefix=' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--datadir=/usr/share' '--includedir=/usr/include' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--prefix=/usr' '--sysconfdir=/etc' '--localstatedir=/var' '--libexecdir=/usr/libexec' '--libdir=/usr/lib' '--with-zlib' '--with-math' '--with-user=netdata' '--disable-dependency-tracking' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'CFLAGS=-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection' 'LDFLAGS=-Wl,-z,relro  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld' 'CXXFLAGS=-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection' 'PKG_CONFIG_PATH=:/usr/lib/pkgconfig:/usr/share/pkgconfig'
Install type: binpkg-rpm
    Binary architecture: x86_64
    Packaging distro:
Features:
    dbengine:                   YES
    Native HTTPS:               YES
    Netdata Cloud:              YES
    ACLK Next Generation:       YES
    ACLK-NG New Cloud Protocol: YES
    ACLK Legacy:                NO
    TLS Host Verification:      YES
    Machine Learning:           YES
    Stream Compression:         NO
Libraries:
    protobuf:                YES (system)
    jemalloc:                NO
    JSON-C:                  YES
    libcap:                  NO
    libcrypto:               YES
    libm:                    YES
    tcalloc:                 NO
    zlib:                    YES
Plugins:
    apps:                    YES
    cgroup Network Tracking: YES
    CUPS:                    YES
    EBPF:                    YES
    IPMI:                    YES
    NFACCT:                  NO
    perf:                    YES
    slabinfo:                YES
    Xen:                     NO
    Xen VBD Error Tracking:  NO
Exporters:
    AWS Kinesis:             NO
    GCP PubSub:              NO
    MongoDB:                 NO
    Prometheus Remote Write: YES
# cat /etc/os-release
NAME="Oracle Linux Server"
VERSION="8.6"
ID="ol"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="8.6"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Oracle Linux Server 8.6"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:oracle:linux:8:6:server"
HOME_URL="https://linux.oracle.com/"
BUG_REPORT_URL="https://bugzilla.oracle.com/"
# uname -a
Linux db01.dba.prod 5.4.17-2136.307.3.1.el8uek.x86_64 #2 SMP Mon May 9 17:29:47 PDT 2022 x86_64 x86_64 x86_64 GNU/Linux
# cat ebpf.d.conf | grep -v '^#'
[global]
    ebpf load mode = entry
    apps = no
    cgroups = no
    update every = 5
    pid table size = 32768
    btf path = /sys/kernel/btf/

[ebpf programs]
    cachestat = no
    dcstat = no
    disk = no
    fd = no
    filesystem = no
    hardirq = no
    mdflush = no
    mount = no
    oomkill = no
    process = no
    shm = no
    socket = no
    softirq = no
    sync = no
    swap = no
    vfs = no
    network connections = yes
# cat ebpf.d/network.conf | grep -v '^#'
[global]
    bandwidth table size = 16384
    ipv4 connection table size = 16384
    ipv6 connection table size = 16384
    udp connection table size = 4096
    ebpf type format = auto
    ebpf co-re tracing = trampoline

[network connections]
    maximum dimensions = 50
    resolve hostnames = no
    resolve service names = no
    ports = *
    ips = !127.0.0.1/8 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 fc00::/7 !::1/128
    hostnames = *

[service name]
    19999 = Netdata

btw, I did validate compatibility:

bash check-kernel-config.sh
[no output]
# mount -t debugfs nodev /sys/kernel/debug
mount: /sys/kernel/debug: nodev already mounted or mount point busy.

# mount -t tracefs nodev /sys/kernel/tracing
mount: /sys/kernel/tracing: nodev already mounted or mount point busy.

Hello @mjtice ,

I will build a VM to verify what is happening with this specific kernel version (5.4 family), because we are not having issues like this when we run on newer kernels. The specific error that you are reporting:

open or load BPF file for thread socket (errno 22, Invalid argument)

can be due:

  • libbpf passing a wrong argument for this kernel version.
  • This specific kernel has different arguments
  • Plugin did not detect properly btf

I will return with answers and the solution as soon I have a complete vision about the issue.

Best regards!

Hello @mjtice ,

I tested on kernel 5.4.210 compiled from scratch and I did not observe any error, so after this I got libvirt VM Oracle 8 and I saw that Oracle Linux is enabling BTF by default, and plugin is not identifying properly the environment when it tries to load the eBPF programs.

Today I was busy with a WMI plugin, but tomorrow I will start my day working with this. I will bring a new update for you tomorrow.

Best regards!

Hello @mjtice ,

This PR Fix oracle linux (eBPF plugin) by thiagoftsm · Pull Request #13935 · netdata/netdata · GitHub is going to fix the issue you reported.

Thank you very much!

Best regards!

Thanks for the quick turn around on this, @Thiago_Marques_0 !

1 Like

PR was merged, next night you will have a Netdata nightly version with the issue fixed, and this month we will also release a new stable version with this.

I apologize for the issues I created, and be free to ping us whenever you need.

Best regards!