ebpf.plugin error

Help
1

The ebpf.plugin generate 0 charts.

One empty chart is mount (eBPF)

the ebpf.plugin throws an error when i start it manually as netdata. Apparently a malloc error.
My call was
/usr/libexec/netdata/plugins.d$ ./ebpf.plugin >/tmp/x 2>&1
Result with error at the end:

CHART mem.cachestat_misses '' 'Files out of page cache' 'misses/s' 'page_cache' '' 'line' 21103 5 '' 'ebpf.plugin' 'cachestat'
DIMENSION miss miss absolute 1 1
malloc(): invalid size (unsorted)

Only Errors at the beginning reported are:

2023-04-02 10:06:35:  INFO  : MAIN : Does not have a configuration file inside `/etc/netdata/ebpf.d.conf. It will try to load stock file.
2023-04-02 10:06:35:  INFO  : MAIN : Cannot read process groups configuration file '/etc/netdata/apps_groups.conf'. Will try '/usr/lib/netdata/conf.d/apps_groups.conf'
2023-04-02 10:06:35:  ERROR : MAIN : PROCFILE: Cannot open file '/proc/1557603/status' (errno 2, No such file or directory)
2023-04-02 10:06:35:  ERROR : MAIN : Cannot open /proc/1557603/status

i could provide the wholw log, if required.

Version: netdata v1.38.0-333-nightly
Debian Bullseye
Memory: RAM: total: 62.8 GiB used: 11.76 GiB (18.7%)
How do I debug this?

2

Hello @Bernd ,

eBPF plugin depends of some kernel options to run properly. Some distributions like Alpine does not compiile eBPF syscall, for example.

So let us debug your system to confirm what is happening:

  1. Are you running on X86_64?
  2. How did you install netdata on your system?
  3. I tested a PR (Update branch by thiagoftsm · Pull Request #39 · netdata/ebpf-co-re · GitHub) on Debian 11 today and I observed that I was running kernel 5.10.0-21-amd64 . Are you running the same kernel? Or did you compile your own kernel?
  4. What is the output of command zgrep "BPF" /proc/config.gz ?
    5 . Can you send the whole output of /tmp/x? I want to check all details to be sure I am not missing nothing.
  5. Do you have any core dump on your system for eBPF.plugin? I already have a PR (New eBPF option by thiagoftsm · Pull Request #14691 · netdata/netdata · GitHub) where I am addressing some errors like invalid size, but I am not sure if we are talking about the same error.

After you answer these answers we will have conditions to help you more.

Best regards!

3
  1. Yes
  2. ii netdata 1.38.0-333-nightly amd64
    ii netdata-repo-edge 2-1 all
  3. 5.10.0-21-amd64 #1 SMP Debian 5.10.162-1 (2023-01-21) x86_64 GNU/Linux
    4.File not found. But: zgrep “BPF” /boot/config-5.10.0-21-amd64
    CONFIG_CGROUP_BPF=y
    CONFIG_BPF=y
    CONFIG_BPF_LSM=y
    CONFIG_BPF_SYSCALL=y
    CONFIG_ARCH_WANT_DEFAULT_BPF_JIT=y

CONFIG_BPF_JIT_ALWAYS_ON is not set

CONFIG_BPF_JIT_DEFAULT_ON=y
CONFIG_BPF_UNPRIV_DEFAULT_OFF=y

CONFIG_BPF_PRELOAD is not set

CONFIG_IPV6_SEG6_BPF=y
CONFIG_NETFILTER_XT_MATCH_BPF=m

CONFIG_BPFILTER is not set

CONFIG_NET_CLS_BPF=m
CONFIG_NET_ACT_BPF=m
CONFIG_BPF_JIT=y
CONFIG_BPF_STREAM_PARSER=y
CONFIG_LWTUNNEL_BPF=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_BPF_EVENTS=y
CONFIG_TEST_BPF=m

5. /tmp/x

2023-04-02 10:06:35:  INFO  : MAIN : Does not have a configuration file inside `/etc/netdata/ebpf.d.conf. It will try to load stock file.
2023-04-02 10:06:35:  INFO  : MAIN : Cannot read process groups configuration file '/etc/netdata/apps_groups.conf'. Will try '/usr/lib/netdata/conf.d/apps_groups.conf'
2023-04-02 10:06:35:  ERROR : MAIN : PROCFILE: Cannot open file '/proc/1557603/status' (errno 2, No such file or directory)
2023-04-02 10:06:35:  ERROR : MAIN : Cannot open /proc/1557603/status
2023-04-02 10:06:35: ebpf.plugin INFO  : MAIN : The range/value (n.n.n.n) is inside the range/value (n.n.n.n) already inserted, it will be ignored.
2023-04-02 10:06:35: ebpf.plugin INFO  : MAIN : The range/value (No text representation) is inside the range/value (No text representation) already inserted, it will be ignored.
2023-04-02 10:06:35: ebpf.plugin INFO  : EBPF PROCESS : thread created with task id 1558293
2023-04-02 10:06:35: ebpf.plugin INFO  : EBPF SWAP : thread created with task id 1558296
2023-04-02 10:06:35: ebpf.plugin INFO  : EBPF SWAP : set name of thread 1558296 to EBPF SWAP
2023-04-02 10:06:35: ebpf.plugin INFO  : EBPF CACHESTAT : thread created with task id 1558294
2023-04-02 10:06:35: ebpf.plugin INFO  : EBPF CACHESTAT : set name of thread 1558294 to EBPF CACHESTAT
2023-04-02 10:06:35: ebpf.plugin INFO  : EBPF FD : thread created with task id 1558298
2023-04-02 10:06:35: ebpf.plugin INFO  : EBPF FD : set name of thread 1558298 to EBPF FD
2023-04-02 10:06:35: ebpf.plugin INFO  : EBPF PROCESS : set name of thread 1558293 to EBPF PROCESS
2023-04-02 10:06:35: ebpf.plugin INFO  : EBPF HARDIRQ : thread created with task id 1558299
2023-04-02 10:06:35: ebpf.plugin INFO  : EBPF HARDIRQ : set name of thread 1558299 to EBPF HARDIRQ
2023-04-02 10:06:35: ebpf.plugin INFO  : EBPF OOMKILL : thread created with task id 1558301
2023-04-02 10:06:35: ebpf.plugin INFO  : EBPF OOMKILL : set name of thread 1558301 to EBPF OOMKILL
2023-04-02 10:06:35: ebpf.plugin INFO  : EBPF MOUNT : thread created with task id 1558297
2023-04-02 10:06:35: ebpf.plugin INFO  : EBPF MOUNT : set name of thread 1558297 to EBPF MOUNT
2023-04-02 10:06:35: ebpf.plugin INFO  : EBPF SOFTIRQ : thread created with task id 1558300
2023-04-02 10:06:35: ebpf.plugin INFO  : EBPF SOFTIRQ : set name of thread 1558300 to EBPF SOFTIRQ
2023-04-02 10:06:35: ebpf.plugin INFO  : EBPF SHM : thread created with task id 1558302
2023-04-02 10:06:35: ebpf.plugin INFO  : EBPF SHM : set name of thread 1558302 to EBPF SHM
2023-04-02 10:06:35: ebpf.plugin INFO  : EBPF SYNC : thread created with task id 1558295
2023-04-02 10:06:35: ebpf.plugin INFO  : EBPF SYNC : set name of thread 1558295 to EBPF SYNC
CHART system.process_thread '' 'Start process' 'calls/s' 'processes' '' 'line' 21002 5 '' 'ebpf.plugin' 'process'
DIMENSION _do_fork process absolute 1 1
DIMENSION sys_clone thread absolute 1 1
DIMENSION _do_fork process absolute 1 1
DIMENSION sys_clone thread absolute 1 1
CHART system.exit '' 'Exit process' 'calls/s' 'processes' '' 'line' 21003 5 '' 'ebpf.plugin' 'process'
DIMENSION do_exit process absolute 1 1
DIMENSION release_task task absolute 1 1
CHART system.process_status '' 'Process not closed' 'difference' 'processes' '' line 21004 5 '' 'ebpf.plugin' 'process'
DIMENSION process '' absolute 1 1
DIMENSION zombie '' absolute 1 1
CHART netdata.ebpf_threads '' 'Threads info.' 'threads' 'ebpf' '' 'line' 140000 5 '' 'ebpf.plugin' 'process'
DIMENSION total total absolute 1 1
DIMENSION running running absolute 1 1
CHART netdata.ebpf_load_methods '' 'Load info.' 'methods' 'ebpf' '' 'line' 140001 5 '' 'ebpf.plugin' 'process'
DIMENSION legacy legacy absolute 1 1
DIMENSION co-re co-re absolute 1 1
CHART netdata.ebpf_kernel_memory '' 'Memory allocated for hash tables.' 'bytes' 'ebpf' '' 'line' 140002 5 '' 'ebpf.plugin' 'process'
DIMENSION memory_locked memory_locked absolute 1 1
CHART netdata.ebpf_hash_tables_count '' 'Number of hash tables loaded.' 'hash tables' 'ebpf' '' 'line' 140003 5 '' 'ebpf.plugin' 'process'
DIMENSION hash_table hash_table absolute 1 1
CHART system.softirq_latency '' 'Software IRQ latency' 'milliseconds' 'softirqs' '' 'stacked' 951 5 '' 'ebpf.plugin' 'softirq'
2023-04-02 10:06:35: ebpf.plugin INFO  : EBPF OOMKILL : thread with task id 1558301 finished
DIMENSION HI HI incremental 1 1
DIMENSION TIMER TIMER incremental 1 1
DIMENSION NET_TX NET_TX incremental 1 1
DIMENSION NET_RX NET_RX incremental 1 1
DIMENSION BLOCK BLOCK incremental 1 1
DIMENSION IRQ_POLL IRQ_POLL incremental 1 1
DIMENSION TASKLET TASKLET incremental 1 1
DIMENSION SCHED SCHED incremental 1 1
DIMENSION HRTIMER HRTIMER incremental 1 1
DIMENSION RCU RCU incremental 1 1
CHART system.hardirq_latency '' 'Hardware IRQ latency' 'milliseconds' 'interrupts' '' 'stacked' 2900 5 '' 'ebpf.plugin' 'hardirq'
DIMENSION apic_thermal apic_thermal incremental 1 1
DIMENSION apic_threshold apic_threshold incremental 1 1
DIMENSION apic_error apic_error incremental 1 1
DIMENSION apic_deferred_error apic_deferred_error incremental 1 1
DIMENSION apic_spurious apic_spurious incremental 1 1
DIMENSION func_call func_call incremental 1 1
DIMENSION func_call_single func_call_single incremental 1 1
DIMENSION reschedule reschedule incremental 1 1
DIMENSION local_timer local_timer incremental 1 1
DIMENSION irq_work irq_work incremental 1 1
DIMENSION x86_platform_ipi x86_platform_ipi incremental 1 1
CHART mem.file_sync '' 'Monitor calls for <code>fsync(2)</code> and <code>fdatasync(2)</code>.' 'calls/s' 'synchronization (eBPF)' '' 'line' 21300 5 '' 'ebpf.plugin' 'sync'
DIMENSION fsync fsync incremental 1 1
DIMENSION fdatasync fdatasync incremental 1 1
CHART mem.memory_map '' 'Monitor calls for <code>msync(2)</code>.' 'calls/s' 'synchronization (eBPF)' '' 'line' 21301 5 '' 'ebpf.plugin' 'sync'
DIMENSION msync msync incremental 1 1
CHART mem.sync '' 'Monitor calls for <code>sync(2)</code> and <code>syncfs(2)</code>.' 'calls/s' 'synchronization (eBPF)' '' 'line' 21302 5 '' 'ebpf.plugin' 'sync'
DIMENSION sync sync incremental 1 1
DIMENSION syncfs syncfs incremental 1 1
CHART mem.file_segment '' 'Monitor calls for <code>sync_file_range(2)</code>.' 'calls/s' 'synchronization (eBPF)' '' 'line' 21303 5 '' 'ebpf.plugin' 'sync'
DIMENSION sync_file_range sync_file_range incremental 1 1
CHART system.shared_memory_calls '' 'Calls to shared memory system calls' 'calls/s' 'ipc shared memory' '' 'line' 1207 5 '' 'ebpf.plugin' 'shm'
DIMENSION get get incremental 1 1
DIMENSION at at incremental 1 1
DIMENSION dt dt incremental 1 1
DIMENSION ctl ctl incremental 1 1
CHART system.swapcalls '' 'Calls to access swap memory' 'calls/s' 'swap' '' 'line' 202 5 '' 'ebpf.plugin' 'swap'
DIMENSION read read incremental 1 1
DIMENSION write write incremental 1 1
CHART mount_points.call '' 'Calls to mount and umount syscalls' 'calls/s' 'mount (eBPF)' '' 'line' 2190 5 '' 'ebpf.plugin' 'mount'
DIMENSION mount mount incremental 1 1
DIMENSION umount umount incremental 1 1
CHART mount_points.error '' 'Errors to mount and umount file systems' 'calls/s' 'mount (eBPF)' '' 'line' 2191 5 '' 'ebpf.plugin' 'mount'
DIMENSION mount mount incremental 1 1
DIMENSION umount umount incremental 1 1
CHART mem.cachestat_ratio '' 'Hit ratio' '%' 'page_cache' '' 'line' 21100 5 '' 'ebpf.plugin' 'cachestat'
DIMENSION ratio ratio absolute 1 1
CHART mem.cachestat_dirties '' 'Number of dirty pages' 'pages/s' 'page_cache' '' 'line' 21101 5 '' 'ebpf.plugin' 'cachestat'
DIMENSION dirty dirty incremental 1 1
CHART mem.cachestat_hits '' 'Number of accessed files' 'hits/s' 'page_cache' '' 'line' 21102 5 '' 'ebpf.plugin' 'cachestat'
DIMENSION hit hit absolute 1 1
CHART mem.cachestat_misses '' 'Files out of page cache' 'misses/s' 'page_cache' '' 'line' 21103 5 '' 'ebpf.plugin' 'cachestat'
DIMENSION miss miss absolute 1 1
malloc(): invalid size (unsorted)

Hope I did not copy paste error.
Thank you for your care.

4

Hello @Bernd ,

Thank you very much for your answers, because they helped a lot.
I tried to replicate the issue, but I could not, on the other hand, another user in discord also reported an issue like this on his VMs and after a debug section today, I could identify the issue and the first PR fixing the issue is already ready Fix HARDIRQ on Debian family by thiagoftsm · Pull Request #330 · netdata/kernel-collector · GitHub. I am only finishing some tests to be sure I did not miss nothing.

Best regards!

1 Like
5

A quick update, after to merge the necessary PRs, I made two new releases and now I am working to bring the final fix.

6

No hurry, thank you for the update.
The longer I use netdata Agent and also the cloud version professionally, the more certain I am that I have made the right choice.

2 Likes
7

I am really happy to read these words. :slight_smile:
I could make the two released I needed yesterday, and the final PR eBPF bug fixes by thiagoftsm · Pull Request #14869 · netdata/netdata · GitHub is working n progress. The final PR will be merged soon.

Best regards!

8

I can confirm that ebpf is working now