eBPF not working

tjk · September 21, 2021, 5:42pm

Hey All,

I’ve got netdata running on some test machines, but I can’t get the eBPF data reporting. I have it enabled in netdata.conf and in ebpf.d.conf with disk = yes.

But I’m not seeing any data locally or via netdata cloud, no eBPF on the bottom right hand side or disk latency stats are being reported.

Where can I start to troubleshoot this?

Edit: Running * v1.31.0-279-nightly, latest Ubuntu 18.04 inside a VMware VM.

andrewm4894 · September 21, 2021, 8:35pm

Hello @tjk welcome!

what do you see if you run

grep 'ebpf' /var/log/netdata/error.log

maybe try it just a few mins after a netdata restart too perhaps.

That might show some info and maybe our eBPF expert @Thiago_Marques_0 might be able to help.

tjk · September 21, 2021, 8:41pm

root@/var/lib/netdata# systemctl restart netdata

root@/var/lib/netdata# grep 'ebpf' /var/log/netdata/error.log

2021-09-21 17:41:29: netdata INFO : PLUGINSD[ **ebpf** ] : thread created with task id 4628

2021-09-21 17:41:29: netdata INFO : PLUGINSD[ **ebpf** ] : set name of thread 4628 to PLUGINSD[ **ebpf** ]

2021-09-21 17:41:29: netdata INFO : PLUGINSD[ **ebpf** ] : 2021-09-21 17:41:29: tc-qos-helper.sh: WARNING: FireQoS is not installed on this system. Use FireQoS to apply traffic QoS and expose the class names to netdata. Check https://github.com/netdata/netdata/tree/master/collectors/tc.plugin#tcplugin

connected to '/usr/libexec/netdata/plugins.d/ **ebpf** .plugin' running on pid 4647

2021-09-21 17:41:29: INFO : MAIN : Does not have a configuration file inside `/etc/netdata/ **ebpf** .d.conf. It will try to load stock file.

2021-09-21 17:41:29: **ebpf** .plugin INFO : EBPF PROCESS : thread created with task id 4787

2021-09-21 17:41:29: **ebpf** .plugin INFO : EBPF PROCESS : set name of thread 4787 to EBPF PROCESS

2021-09-21 17:41:29: **ebpf** .plugin INFO : EBPF SYNC : thread created with task id 4790

2021-09-21 17:41:29: **ebpf** .plugin INFO : EBPF SYNC : set name of thread 4790 to EBPF SYNC

2021-09-21 17:41:29: **ebpf** .plugin INFO : EBPF CACHESTAT : thread created with task id 4789

2021-09-21 17:41:29: **ebpf** .plugin INFO : EBPF CACHESTAT : set name of thread 4789 to EBPF CACHESTAT

2021-09-21 17:41:29: **ebpf** .plugin INFO : EBPF CACHESTAT : thread with task id 4789 finished

2021-09-21 17:41:29: **ebpf** .plugin INFO : EBPF SWAP : thread created with task id 4792

2021-09-21 17:41:29: **ebpf** .plugin INFO : EBPF SWAP : set name of thread 4792 to EBPF SWAP

2021-09-21 17:41:29: **ebpf** .plugin INFO : EBPF SWAP : thread with task id 4792 finished

2021-09-21 17:41:29: **ebpf** .plugin INFO : EBPF DCSTAT : thread created with task id 4791

2021-09-21 17:41:29: **ebpf** .plugin INFO : EBPF DCSTAT : set name of thread 4791 to EBPF DCSTAT

2021-09-21 17:41:29: **ebpf** .plugin INFO : EBPF VFS : thread created with task id 47932021-09-21 17:41:29: go.d ERROR: prometheus[php-fpm_exporter_local] check failed

2021-09-21 17:41:29: **ebpf** .plugin INFO : EBPF VFS : set name of thread 4793 to EBPF VFS

2021-09-21 17:41:29: **ebpf** .plugin INFO : EBPF SOCKET : thread created with task id 4788

2021-09-21 17:41:29: **ebpf** .plugin INFO : EBPF SOCKET : set name of thread 4788 to EBPF SOCKET

2021-09-21 17:41:29: **ebpf** .plugin INFO : EBPF FILESYSTEM : thread created with task id 4794

2021-09-21 17:41:29: **ebpf** .plugin INFO : EBPF FILESYSTEM : set name of thread 4794 to EBPF FILESYSTEM

2021-09-21 17:41:29: **ebpf** .plugin INFO : EBPF FILESYSTEM : thread with task id 4794 finished

2021-09-21 17:41:29: **ebpf** .plugin INFO : EBPF MOUNT : thread created with task id 4796

2021-09-21 17:41:29: **ebpf** .plugin INFO : EBPF DISK : thread created with task id 4795

2021-09-21 17:41:29: **ebpf** .plugin INFO : EBPF DISK : set name of thread 4795 to EBPF DISK

libbpf: failed to load object '/usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _sync.4.16.o'

2021-09-21 17:41:29: **ebpf** .plugin ERROR : EBPF SYNC : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _sync.4.16.o

2021-09-21 17:41:29: **ebpf** .plugin INFO : EBPF SYNC : thread with task id 4790 finished

2021-09-21 17:41:29: **ebpf** .plugin INFO : EBPF DISK : thread with task id 4795 finished

2021-09-21 17:41:29: **ebpf** .plugin INFO : EBPF SOCKET : Name resolution is disabled, collector will not parser "hostnames" list.

2021-09-21 17:41:29: **ebpf** .plugin INFO : EBPF MOUNT : 2021-09-21 17:41:29: go.d ERROR: prometheus[hello_sense_exporter_local] check failed

2021-09-21 17:41:29: **ebpf** .plugin INFO : EBPF FD : thread created with task id 4797

2021-09-21 17:41:29: **ebpf** .plugin INFO : EBPF FD : set name of thread 4797 to EBPF FD

2021-09-21 17:41:29: **ebpf** .plugin INFO : EBPF SOCKET : 2021-09-21 17:41:29: go.d ERROR: prometheus[azure_resources_exporter_local] check failed

2021-09-21 17:41:29: **ebpf** .plugin INFO : EBPF SOCKET : The range/value (127.0.0.1/8) is inside the range/value (127.0.0.1/8) already inserted, it will be ignored.

2021-09-21 17:41:29: **ebpf** .plugin INFO : EBPF SOCKET : The range/value (10.0.0.0/8) is inside the range/value (10.0.0.0/8) already inserted, it will be ignored.

2021-09-21 17:41:29: **ebpf** .plugin INFO : EBPF SOCKET : The range/value (172.16.0.0/12) is inside the range/value (172.16.0.0/12) already inserted, it will be ignored.

2021-09-21 17:41:29: **ebpf** .plugin INFO : EBPF SOCKET : The range/value (192.168.0.0/16) is inside the range/value (192.168.0.0/16) already inserted, it will be ignored.

2021-09-21 17:41:29: **ebpf** .plugin INFO : EBPF SOCKET : The range/value (fc00::/7) is inside the range/value (fc00::/7) already inserted, it will be ignored.

2021-09-21 17:41:29: **ebpf** .plugin INFO : EBPF SOCKET : The range/value (::1/128) is inside the range/value (::1/128) already inserted, it will be ignored.

2021-09-21 17:41:29: **ebpf** .plugin INFO : EBPF HARDIRQ : thread created with task id 4798

2021-09-21 17:41:29: **ebpf** .plugin INFO : EBPF HARDIRQ : set name of thread 4798 to EBPF HARDIRQ

libbpf: failed to load object '/usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _mount.4.16.o'

2021-09-21 17:41:29: **ebpf** .plugin ERROR : EBPF MOUNT : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _mount.4.16.o

2021-09-21 17:41:29: **ebpf** .plugin INFO : EBPF MOUNT : thread with task id 4796 finished

2021-09-21 17:41:29: **ebpf** .plugin INFO : EBPF SOFTIRQ : thread created with task id 4799

2021-09-21 17:41:29: **ebpf** .plugin INFO : EBPF SOFTIRQ : set name of thread 4799 to EBPF SOFTIRQ

libbpf: failed to load object '/usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _process.4.16.o'

2021-09-21 17:41:29: **ebpf** .plugin ERROR : EBPF PROCESS : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _process.4.16.o

2021-09-21 17:41:29: **ebpf** .plugin INFO : EBPF OOMKILL : thread created with task id 4802

2021-09-21 17:41:29: **ebpf** .plugin INFO : EBPF OOMKILL : set name of thread 4802 to EBPF OOMKILL

2021-09-21 17:41:29: **ebpf** .plugin INFO : SOFTIRQ KERNEL : thread created with task id 4809

2021-09-21 17:41:29: **ebpf** .plugin INFO : SOFTIRQ KERNEL : set name of thread 4809 to SOFTIRQ KERNEL

libbpf: failed to load object '/usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _fd.4.16.o'

2021-09-21 17:41:29: **ebpf** .plugin ERROR : EBPF FD : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _fd.4.16.o

2021-09-21 17:41:29: **ebpf** .plugin INFO : EBPF FD : thread with task id 4797 finished

libbpf: failed to load object '/usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _vfs.4.16.o'

2021-09-21 17:41:29: **ebpf** .plugin ERROR : EBPF VFS : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _vfs.4.16.o

2021-09-21 17:41:29: **ebpf** .plugin INFO : EBPF VFS : 2021-09-21 17:41:29: go.d ERROR: prometheus[connection_status_exporter_local] check failed

2021-09-21 17:41:29: **ebpf** .plugin INFO : HARDIRQ KERNEL : thread created with task id 4843

2021-09-21 17:41:29: **ebpf** .plugin INFO : HARDIRQ KERNEL : 2021-09-21 17:41:29: go.d ERROR: prometheus[vespa_engine_exporter_local] check failed

2021-09-21 17:41:29: go.d ERROR: prometheus[ **ebpf** _exporter_local] Get "http://127.0.0.1:9435/metrics": dial tcp 127.0.0.1:9435: connect: connection refused

2021-09-21 17:41:29: go.d ERROR: prometheus[ **ebpf** _exporter_local] check failed

2021-09-21 17:41:29: **ebpf** .plugin INFO : EBPF DCSTAT : thread with task id 4791 finished

libbpf: failed to load object '/usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _socket.4.16.o'

2021-09-21 17:41:30: **ebpf** .plugin ERROR : EBPF SOCKET : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _socket.4.16.o

2021-09-21 17:41:30: **ebpf** .plugin INFO : EBPF SOCKET : thread with task id 4788 finished

2021-09-21 17:41:31: **ebpf** .plugin INFO : EBPF PROCESS : thread with task id 4787 finished

2021-09-21 17:46:55: netdata INFO : PLUGINSD : stopping plugin thread: plugin: **ebpf**

2021-09-21 17:46:55: netdata INFO : PLUGINSD[ **ebpf** ] : data collection thread exiting

2021-09-21 17:46:55: netdata INFO : PLUGINSD[ **ebpf** ] : killing child process pid 4647

2021-09-21 17:46:55: netdata INFO : PLUGINSD[ **ebpf** ] : waiting for child process pid 4647 to exit...

2021-09-21 17:46:55: **ebpf** .plugin INFO : SOFTIRQ KERNEL : thread with task id 4809 finished

2021-09-21 17:46:55: **ebpf** .plugin INFO : HARDIRQ KERNEL : thread with task id 4843 finished

2021-09-21 17:49:26: netdata INFO : PLUGINSD[ **ebpf** ] : thread created with task id 5591

2021-09-21 17:49:26: netdata INFO : PLUGINSD[ **ebpf** ] : set name of thread 5591 to PLUGINSD[ **ebpf** ]

2021-09-21 17:49:26: netdata INFO : PLUGINSD[ **ebpf** ] : connected to '/usr/libexec/netdata/plugins.d/ **ebpf** .plugin' running on pid 5608

2021-09-21 17:49:26: **ebpf** .plugin INFO : EBPF SOCKET : thread created with task id 5734

2021-09-21 17:49:26: **ebpf** .plugin INFO : EBPF SOCKET : set name of thread 5734 to EBPF SOCKET

2021-09-21 17:49:26: **ebpf** .plugin INFO : EBPF SYNC : thread created with task id 5736

2021-09-21 17:49:26: **ebpf** .plugin INFO : EBPF SYNC : set name of thread 5736 to EBPF SYNC

2021-09-21 17:49:26: **ebpf** .plugin INFO : EBPF SOCKET : Name resolution is disabled, collector will not parser "hostnames" list.

2021-09-21 17:49:26: **ebpf** .plugin INFO : EBPF CACHESTAT : thread created with task id 5735

2021-09-21 17:49:26: **ebpf** .plugin INFO : EBPF CACHESTAT : set name of thread 5735 to EBPF CACHESTAT

2021-09-21 17:49:26: **ebpf** .plugin INFO : EBPF FILESYSTEM : thread created with task id 5740

2021-09-21 17:49:26: **ebpf** .plugin INFO : EBPF DISK : thread created with task id 5741

2021-09-21 17:49:26: **ebpf** .plugin INFO : EBPF SOCKET : 2021-09-21 17:49:26: go.d ERROR: prometheus[fluentd_exporter_local] check failed

2021-09-21 17:49:26: **ebpf** .plugin INFO : EBPF SOCKET : The range/value (127.0.0.1/8) is inside the range/value (127.0.0.1/8) already inserted, it will be ignored.

2021-09-21 17:49:26: **ebpf** .plugin INFO : EBPF SOCKET : The range/value (10.0.0.0/8) is inside the range/value (10.0.0.0/8) already inserted, it will be ignored.

2021-09-21 17:49:26: **ebpf** .plugin INFO : EBPF SOCKET : The range/value (172.16.0.0/12) is inside the range/value (172.16.0.0/12) already inserted, it will be ignored.

2021-09-21 17:49:26: **ebpf** .plugin INFO : EBPF SOCKET : The range/value (192.168.0.0/16) is inside the range/value (192.168.0.0/16) already inserted, it will be ignored.

2021-09-21 17:49:26: **ebpf** .plugin INFO : EBPF SOCKET : The range/value (fc00::/7) is inside the range/value (fc00::/7) already inserted, it will be ignored.

2021-09-21 17:49:26: **ebpf** .plugin INFO : EBPF SOCKET : The range/value (::1/128) is inside the range/value (::1/128) already inserted, it will be ignored.

2021-09-21 17:49:26: **ebpf** .plugin INFO : EBPF DISK : 2021-09-21 17:49:26: go.d ERROR: prometheus[mailexporter_local] check failed

2021-09-21 17:49:26: **ebpf** .plugin INFO : EBPF DCSTAT : thread created with task id 5737

2021-09-21 17:49:26: **ebpf** .plugin INFO : EBPF DCSTAT : set name of thread 5737 to EBPF DCSTAT

2021-09-21 17:49:26: **ebpf** .plugin INFO : EBPF SWAP : 2021-09-21 17:49:26: go.d ERROR: prometheus[allas_local] check failed

2021-09-21 17:49:26: **ebpf** .plugin INFO : EBPF SWAP : set name of thread 5738 to EBPF SWAP

2021-09-21 17:49:26: **ebpf** .plugin INFO : EBPF SWAP : thread with task id 5738 finished

2021-09-21 17:49:26: **ebpf** .plugin INFO : EBPF CACHESTAT : thread with task id 5735 finished

2021-09-21 17:49:26: **ebpf** .plugin INFO : EBPF FILESYSTEM : set name of thread 5740 to EBPF FILESYSTEM

2021-09-21 17:49:26: **ebpf** .plugin INFO : EBPF VFS : thread created with task id 5739

2021-09-21 17:49:26: **ebpf** .plugin INFO : EBPF VFS : set name of thread 5739 to EBPF VFS

2021-09-21 17:49:26: **ebpf** .plugin INFO : EBPF FD : thread created with task id 5744

2021-09-21 17:49:26: **ebpf** .plugin INFO : EBPF FD : set name of thread 5744 to EBPF FD

libbpf: failed to load object '/usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _sync.4.16.o'

2021-09-21 17:49:26: **ebpf** .plugin INFO : EBPF PROCESS : 2021-09-21 17:49:26: go.d ERROR: prometheus[network_ups_tools_exporter_local] Get "http://127.0.0.1:9230/metrics": dial tcp 127.0.0.1:9230: connect: connection refused

2021-09-21 17:49:26: **ebpf** .plugin INFO : EBPF PROCESS : set name of thread 5733 to EBPF PROCESS

2021-09-21 17:49:26: **ebpf** .plugin ERROR : EBPF SYNC : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _sync.4.16.o

2021-09-21 17:49:26: **ebpf** .plugin INFO : EBPF SYNC : thread with task id 5736 finished

2021-09-21 17:49:26: **ebpf** .plugin INFO : EBPF MOUNT : thread created with task id 5742

2021-09-21 17:49:26: **ebpf** .plugin INFO : EBPF MOUNT : set name of thread 5742 to EBPF MOUNT

2021-09-21 17:49:26: **ebpf** .plugin INFO : EBPF OOMKILL : thread created with task id 5747

2021-09-21 17:49:26: **ebpf** .plugin INFO : EBPF OOMKILL : set name of thread 5747 to EBPF OOMKILL

2021-09-21 17:49:26: **ebpf** .plugin INFO : EBPF SOFTIRQ : thread created with task id 5746

2021-09-21 17:49:26: **ebpf** .plugin INFO : EBPF SOFTIRQ : set name of thread 5746 to EBPF SOFTIRQ

2021-09-21 17:49:26: **ebpf** .plugin INFO : EBPF HARDIRQ : thread created with task id 5745

2021-09-21 17:49:26: **ebpf** .plugin INFO : EBPF HARDIRQ : set name of thread 5745 to EBPF HARDIRQ

libbpf: failed to load object '/usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _mount.4.16.o'

2021-09-21 17:49:26: **ebpf** .plugin ERROR : EBPF MOUNT : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _mount.4.16.o

2021-09-21 17:49:26: **ebpf** .plugin INFO : EBPF MOUNT : thread with task id 5742 finished

2021-09-21 17:49:26: **ebpf** .plugin INFO : SOFTIRQ KERNEL : thread created with task id 5761

2021-09-21 17:49:26: **ebpf** .plugin INFO : SOFTIRQ KERNEL : set name of thread 5761 to SOFTIRQ KERNEL

libbpf: failed to load object '/usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _ext4.4.16.o'

2021-09-21 17:49:26: **ebpf** .plugin ERROR : EBPF FILESYSTEM : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _ext4.4.16.o

2021-09-21 17:49:26: **ebpf** .plugin INFO : EBPF FILESYSTEM : 2021-09-21 17:49:26: go.d ERROR: prometheus[ceph_ceph-mgr_prometheus_plugin_local] check failed

libbpf: failed to load object '/usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _fd.4.16.o'

2021-09-21 17:49:26: **ebpf** .plugin ERROR : EBPF FD : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _fd.4.16.o

2021-09-21 17:49:26: **ebpf** .plugin INFO : EBPF FD : thread with task id 5744 finished

libbpf: failed to load object '/usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _process.4.16.o'

2021-09-21 17:49:26: **ebpf** .plugin ERROR : EBPF PROCESS : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _process.4.16.o

libbpf: failed to load object '/usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _vfs.4.16.o'

2021-09-21 17:49:26: **ebpf** .plugin ERROR : EBPF VFS : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _vfs.4.16.o

2021-09-21 17:49:26: **ebpf** .plugin INFO : EBPF VFS : thread with task id 5739 finished

2021-09-21 17:49:26: **ebpf** .plugin INFO : DISK KERNEL : thread created with task id 5764

2021-09-21 17:49:26: **ebpf** .plugin INFO : DISK KERNEL : set name of thread 5764 to DISK KERNEL

2021-09-21 17:49:26: **ebpf** .plugin INFO : HARDIRQ KERNEL : thread created with task id 5793

2021-09-21 17:49:26: **ebpf** .plugin INFO : HARDIRQ KERNEL : set name of thread 5793 to HARDIRQ KERNEL

2021-09-21 17:49:26: go.d ERROR: prometheus[ **ebpf** _exporter_local] Get "http://127.0.0.1:9435/metrics": dial tcp 127.0.0.1:9435: connect: connection refused

2021-09-21 17:49:26: go.d ERROR: prometheus[ **ebpf** _exporter_local] check failed

2021-09-21 17:49:26: **ebpf** .plugin INFO : EBPF DCSTAT : thread with task id 5737 finished

libbpf: failed to load object '/usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _socket.4.16.o'

2021-09-21 17:49:27: **ebpf** .plugin ERROR : EBPF SOCKET : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _socket.4.16.o

2021-09-21 17:49:27: **ebpf** .plugin INFO : EBPF SOCKET : thread with task id 5734 finished

2021-09-21 17:49:28: **ebpf** .plugin INFO : EBPF PROCESS : thread with task id 5733 finished

2021-09-21 20:36:26: netdata INFO : PLUGINSD : stopping plugin thread: plugin: **ebpf**

2021-09-21 20:36:26: netdata INFO : PLUGINSD[ **ebpf** ] : data collection thread exiting

2021-09-21 20:36:26: netdata INFO : PLUGINSD[ **ebpf** ] : killing child process pid 5608

2021-09-21 20:36:26: netdata INFO : PLUGINSD[ **ebpf** ] : waiting for child process pid 5608 to exit...

2021-09-21 20:36:26: **ebpf** .plugin INFO : DISK KERNEL : thread with task id 5764 finished

2021-09-21 20:36:26: **ebpf** .plugin INFO : SOFTIRQ KERNEL : thread with task id 5761 finished

2021-09-21 20:36:26: **ebpf** .plugin INFO : HARDIRQ KERNEL : thread with task id 5793 finished

2021-09-21 20:38:57: netdata INFO : PLUGINSD[ **ebpf** ] : thread created with task id 16931

2021-09-21 20:38:57: netdata INFO : PLUGINSD[ **ebpf** ] : set name of thread 16931 to PLUGINSD[ **ebpf** ]

2021-09-21 20:38:57: netdata INFO : PLUGINSD[ **ebpf** ] : connected to '/usr/libexec/netdata/plugins.d/ **ebpf** .plugin' running on pid 16935

2021-09-21 20:38:57: **ebpf** .plugin INFO : EBPF PROCESS : thread created with task id 17123

2021-09-21 20:38:57: **ebpf** .plugin INFO : EBPF CACHESTAT : thread created with task id 17125

2021-09-21 20:38:57: **ebpf** .plugin INFO : EBPF DCSTAT : thread created with task id 17127

2021-09-21 20:38:57: **ebpf** .plugin INFO : EBPF SOCKET : thread created with task id 17124

2021-09-21 20:38:57: **ebpf** .plugin INFO : EBPF SOCKET : set name of thread 17124 to EBPF SOCKET

2021-09-21 20:38:57: **ebpf** .plugin INFO : EBPF PROCESS : set name of thread 17123 to EBPF PROCESS

2021-09-21 20:38:57: **ebpf** .plugin INFO : EBPF FILESYSTEM : thread created with task id 17130

2021-09-21 20:38:57: **ebpf** .plugin INFO : EBPF FILESYSTEM : set name of thread 17130 to EBPF FILESYSTEM

2021-09-21 20:38:57: **ebpf** .plugin INFO : EBPF CACHESTAT : set name of thread 17125 to EBPF CACHESTAT

2021-09-21 20:38:57: **ebpf** .plugin INFO : EBPF VFS : thread created with task id 17129

2021-09-21 20:38:57: **ebpf** .plugin INFO : EBPF DCSTAT : set name of thread 17127 to EBPF DCSTAT

2021-09-21 20:38:57: **ebpf** .plugin INFO : EBPF FD : thread created with task id 17133

2021-09-21 20:38:57: **ebpf** .plugin INFO : EBPF FD : set name of thread 17133 to EBPF FD

2021-09-21 20:38:57: **ebpf** .plugin INFO : EBPF SWAP : thread created with task id 17128

2021-09-21 20:38:57: **ebpf** .plugin INFO : EBPF SWAP : set name of thread 17128 to EBPF SWAP

2021-09-21 20:38:57: **ebpf** .plugin INFO : EBPF SOCKET : Name resolution is disabled, collector will not parser "hostnames" list.

2021-09-21 20:38:57: **ebpf** .plugin INFO : EBPF CACHESTAT : thread with task id 17125 finished

2021-09-21 20:38:57: **ebpf** .plugin INFO : EBPF MOUNT : thread created with task id 17132

2021-09-21 20:38:57: **ebpf** .plugin INFO : EBPF MOUNT : set name of thread 17132 to EBPF MOUNT

2021-09-21 20:38:57: **ebpf** .plugin INFO : EBPF DISK : thread created with task id 17131

2021-09-21 20:38:57: **ebpf** .plugin INFO : EBPF DISK : set name of thread 17131 to EBPF DISK

2021-09-21 20:38:57: **ebpf** .plugin INFO : EBPF SYNC : thread created with task id 17126

2021-09-21 20:38:57: **ebpf** .plugin INFO : EBPF SYNC : set name of thread 17126 to EBPF SYNC

2021-09-21 20:38:57: **ebpf** .plugin INFO : EBPF SWAP : thread with task id 17128 finished

2021-09-21 20:38:57: **ebpf** .plugin INFO : EBPF HARDIRQ : thread created with task id 17134

libbpf: failed to load object '/usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _mount.4.16.o'

libbpf: failed to load object '/usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _process.4.16.o'

2021-09-21 20:38:57: **ebpf** .plugin INFO : EBPF HARDIRQ : set name of thread 17134 to EBPF HARDIRQ

2021-09-21 20:38:57: **ebpf** .plugin INFO : EBPF SOFTIRQ : thread created with task id 17135

2021-09-21 20:38:57: **ebpf** .plugin INFO : EBPF SOFTIRQ : set name of thread 17135 to EBPF SOFTIRQ

2021-09-21 20:38:57: **ebpf** .plugin INFO : EBPF VFS : set name of thread 17129 to EBPF VFS

2021-09-21 20:38:57: **ebpf** .plugin INFO : EBPF OOMKILL : thread created with task id 17136

2021-09-21 20:38:57: **ebpf** .plugin INFO : EBPF OOMKILL : set name of thread 17136 to EBPF OOMKILL

2021-09-21 20:38:57: **ebpf** .plugin INFO : EBPF SOCKET : The network value of CIDR 127.0.0.1/8 was updated for 127.0.0.0 .

2021-09-21 20:38:57: **ebpf** .plugin INFO : EBPF SOCKET : The range/value (127.0.0.1/8) is inside the range/value (127.0.0.1/8) already inserted, it will be ignored.

2021-09-21 20:38:57: **ebpf** .plugin INFO : EBPF SOCKET : The range/value (10.0.0.0/8) is inside the range/value (10.0.0.0/8) already inserted, it will be ignored.

2021-09-21 20:38:57: **ebpf** .plugin INFO : EBPF SOCKET : The range/value (172.16.0.0/12) is inside the range/value (172.16.0.0/12) already inserted, it will be ignored.

2021-09-21 20:38:57: **ebpf** .plugin INFO : EBPF SOCKET : The range/value (192.168.0.0/16) is inside the range/value (192.168.0.0/16) already inserted, it will be ignored.

2021-09-21 20:38:57: **ebpf** .plugin INFO : EBPF SOCKET : The range/value (fc00::/7) is inside the range/value (fc00::/7) already inserted, it will be ignored.

2021-09-21 20:38:57: **ebpf** .plugin INFO : EBPF SOCKET : The range/value (::1/128) is inside the range/value (::1/128) already inserted, it will be ignored.

2021-09-21 20:38:57: **ebpf** .plugin ERROR : EBPF PROCESS : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _process.4.16.o

2021-09-21 20:38:57: **ebpf** .plugin ERROR : EBPF MOUNT : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _mount.4.16.o

2021-09-21 20:38:57: **ebpf** .plugin INFO : EBPF MOUNT : thread with task id 17132 finished

libbpf: failed to load object '/usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _fd.4.16.o'

2021-09-21 20:38:57: **ebpf** .plugin ERROR : EBPF FD : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _fd.4.16.o

2021-09-21 20:38:57: **ebpf** .plugin INFO : EBPF FD : thread with task id 17133 finished

libbpf: failed to load object '/usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _sync.4.16.o'

2021-09-21 20:38:57: **ebpf** .plugin ERROR : EBPF SYNC : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _sync.4.16.o

2021-09-21 20:38:57: **ebpf** .plugin INFO : EBPF SYNC : thread with task id 17126 finished

libbpf: failed to load object '/usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _ext4.4.16.o'

2021-09-21 20:38:57: **ebpf** .plugin ERROR : EBPF FILESYSTEM : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _ext4.4.16.o

2021-09-21 20:38:57: **ebpf** .plugin INFO : EBPF FILESYSTEM : thread with task id 17130 finished

2021-09-21 20:38:57: **ebpf** .plugin INFO : SOFTIRQ KERNEL : thread created with task id 17143

2021-09-21 20:38:57: **ebpf** .plugin INFO : SOFTIRQ KERNEL : set name of thread 17143 to SOFTIRQ KERNEL

libbpf: failed to load object '/usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _vfs.4.16.o'

2021-09-21 20:38:57: **ebpf** .plugin ERROR : EBPF VFS : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _vfs.4.16.o

2021-09-21 20:38:57: **ebpf** .plugin INFO : EBPF VFS : thread with task id 17129 finished

2021-09-21 20:38:57: **ebpf** .plugin INFO : HARDIRQ KERNEL : thread created with task id 17158

2021-09-21 20:38:57: **ebpf** .plugin INFO : HARDIRQ KERNEL : set name of thread 17158 to HARDIRQ KERNEL

2021-09-21 20:38:57: **ebpf** .plugin INFO : DISK KERNEL : thread created with task id 17159

2021-09-21 20:38:57: **ebpf** .plugin INFO : DISK KERNEL : set name of thread 17159 to DISK KERNEL

2021-09-21 20:38:57: **ebpf** .plugin INFO : EBPF DCSTAT : thread with task id 17127 finished

libbpf: failed to load object '/usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _socket.4.16.o'

2021-09-21 20:38:57: **ebpf** .plugin ERROR : EBPF SOCKET : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _socket.4.16.o

2021-09-21 20:38:57: **ebpf** .plugin INFO : EBPF SOCKET : thread with task id 17124 finished

2021-09-21 20:38:58: go.d ERROR: prometheus[ **ebpf** _exporter_local] Get "http://127.0.0.1:9435/metrics": dial tcp 127.0.0.1:9435: connect: connection refused

2021-09-21 20:38:58: go.d ERROR: prometheus[ **ebpf** _exporter_local] check failed

2021-09-21 20:38:59: **ebpf** .plugin INFO : EBPF PROCESS : thread with task id 17123 finished

Thiago_Marques_0 · September 21, 2021, 11:54pm

Hello @tjk ,

I remember that Ubuntu 18.04 had some problems when we were using kernel 4.18.x, and it was disabled it in the past, please, run the following commands and verify whether it will run on Ubuntu:

cd /etc/netdata/
./edit-config ebpf.d/ebpf_kernel_reject_list.txt

When your favorite editor opens, please, remove the line inside the configuration file.
After this, try restart Netdata.

Any doubt, please let us know.

Best regards,

Thiago

tjk · September 22, 2021, 12:16am

Thanks, this didn’t work.

In that file was Ubuntu 4.18.0, I removed it and restarted netdata.

Not sure if it helps, but this was working 3 or 4 weeks ago on the same VM. I went back to look at latency stats earlier today and noticed they were not working any longer.

Not sure if an update in the agent over the last couple of weeks broke it or what happened, but it was working.

This was the guide I followed: eBPF monitoring with Netdata | Learn Netdata

And this scripts return null value, saying it is compatible: curl -sSL https://raw.githubusercontent.com/netdata/kernel-collector/master/tools/check-kernel-config.sh | sudo bash

Kernel version - Linux validator3 4.15.0-158-generic #166-Ubuntu SMP Fri Sep 17 19:37:52 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

ilyam8 · September 22, 2021, 7:22am

@Thiago_Marques_0 could be related, there is a bunch of “loading BPF object file failed” loglines

2021-09-21 17:41:29: **ebpf** .plugin ERROR : EBPF SYNC : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _sync.4.16.o
2021-09-21 17:41:29: **ebpf** .plugin ERROR : EBPF MOUNT : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _mount.4.16.o
2021-09-21 17:41:29: **ebpf** .plugin ERROR : EBPF PROCESS : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _process.4.16.o
2021-09-21 17:41:29: **ebpf** .plugin ERROR : EBPF FD : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _fd.4.16.o
2021-09-21 17:41:29: **ebpf** .plugin ERROR : EBPF VFS : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _vfs.4.16.o
2021-09-21 17:41:30: **ebpf** .plugin ERROR : EBPF SOCKET : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _socket.4.16.o
2021-09-21 17:49:26: **ebpf** .plugin ERROR : EBPF SYNC : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _sync.4.16.o
2021-09-21 17:49:26: **ebpf** .plugin ERROR : EBPF MOUNT : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _mount.4.16.o
2021-09-21 17:49:26: **ebpf** .plugin ERROR : EBPF FILESYSTEM : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _ext4.4.16.o
2021-09-21 17:49:26: **ebpf** .plugin ERROR : EBPF FD : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _fd.4.16.o
2021-09-21 17:49:26: **ebpf** .plugin ERROR : EBPF PROCESS : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _process.4.16.o
2021-09-21 17:49:26: **ebpf** .plugin ERROR : EBPF VFS : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _vfs.4.16.o
2021-09-21 17:49:27: **ebpf** .plugin ERROR : EBPF SOCKET : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _socket.4.16.o
2021-09-21 20:38:57: **ebpf** .plugin ERROR : EBPF PROCESS : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _process.4.16.o
2021-09-21 20:38:57: **ebpf** .plugin ERROR : EBPF MOUNT : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _mount.4.16.o
2021-09-21 20:38:57: **ebpf** .plugin ERROR : EBPF FD : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _fd.4.16.o
2021-09-21 20:38:57: **ebpf** .plugin ERROR : EBPF SYNC : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _sync.4.16.o
2021-09-21 20:38:57: **ebpf** .plugin ERROR : EBPF FILESYSTEM : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _ext4.4.16.o
2021-09-21 20:38:57: **ebpf** .plugin ERROR : EBPF VFS : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _vfs.4.16.o
2021-09-21 20:38:57: **ebpf** .plugin ERROR : EBPF SOCKET : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ **ebpf** .d/pnetdata_ **ebpf** _socket.4.16.o

Thiago_Marques_0 · September 23, 2021, 2:28am

Hello @ilyam8 and @tjk ,

I tested now on a VM that I have Ubuntu 18.04, this VM has 3 kernels 5.4, 4.18, and 4.15 . I compiled the latest Netdata code with few modifications that have relationship with PRs I am working. When I compiled, I was running kernel 5.4 and I did not have problems to run eBPF plugin on all these kernels.

@tjk considering what @ilyam8 wrote for us, I have questions for you:

do you have the directory /usr/libexec/netdata/plugins.d/ ebpf.d?
Please, can you stop netdata and go to directory /usr/libexec/netdata/plugins.d/and after this to run ./ebpf.plugin?
Finally, I saw that you are running nightly Netdata. Are you also updating Ubuntu every day? I was 10 days without turn on this VM , and I had to wait more than 1 hour before to recompile the whole Netdata.
Did you have problems between Netdata update?

Best regards!

tjk · September 23, 2021, 5:21pm

Yes, I have this directory, in it are a bunch of pnet* and rnet* files.

Here is the output of ./ebpf.plugin:

root@validator3:/usr/libexec/netdata/plugins.d# ./ebpf.plugin

2021-09-23 13:13:50: INFO : MAIN : Name resolution is disabled, collector will not parser "hostnames" list.

2021-09-23 13:13:50: INFO : MAIN : The network value of CIDR 127.0.0.1/8 was updated for 127.0.0.0 .

2021-09-23 13:13:50: INFO : MAIN : Loaded config file '/etc/netdata/apps_groups.conf'

2021-09-23 13:13:50: ERROR : MAIN : PROCFILE: Cannot open file '/proc/1520/status' (errno 2, No such file or directory)

2021-09-23 13:13:50: ERROR : MAIN : Cannot open /proc/1520/status

2021-09-23 13:13:50: ebpf.plugin INFO : EBPF PROCESS : thread created with task id 22285

2021-09-23 13:13:50: ebpf.plugin INFO : EBPF PROCESS : set name of thread 22285 to EBPF PROCESS

2021-09-23 13:13:50: ebpf.plugin INFO : EBPF SOCKET : thread created with task id 22286

2021-09-23 13:13:50: ebpf.plugin INFO : EBPF CACHESTAT : thread created with task id 22287

2021-09-23 13:13:50: ebpf.plugin INFO : EBPF CACHESTAT : set name of thread 22287 to EBPF CACHESTAT

2021-09-23 13:13:50: ebpf.plugin INFO : EBPF CACHESTAT : thread with task id 22287 finished

2021-09-23 13:13:50: ebpf.plugin INFO : EBPF SOCKET : set name of thread 22286 to EBPF SOCKET

2021-09-23 13:13:50: ebpf.plugin INFO : EBPF SYNC : thread created with task id 22288

2021-09-23 13:13:50: ebpf.plugin INFO : EBPF DCSTAT : thread created with task id 22289

2021-09-23 13:13:50: ebpf.plugin INFO : EBPF DCSTAT : set name of thread 22289 to EBPF DCSTAT

2021-09-23 13:13:50: ebpf.plugin INFO : EBPF SYNC : set name of thread 22288 to EBPF SYNC

2021-09-23 13:13:50: ebpf.plugin INFO : EBPF SOCKET : Name resolution is disabled, collector will not parser "hostnames" list.

2021-09-23 13:13:50: ebpf.plugin INFO : EBPF SOCKET : The network value of CIDR 127.0.0.1/8 was updated for 127.0.0.0 .

2021-09-23 13:13:50: ebpf.plugin INFO : EBPF SOCKET : The range/value (127.0.0.1/8) is inside the range/value (127.0.0.1/8) already inserted, it will be ignored.

2021-09-23 13:13:50: ebpf.plugin INFO : EBPF SOCKET : The range/value (10.0.0.0/8) is inside the range/value (10.0.0.0/8) already inserted, it will be ignored.

2021-09-23 13:13:50: ebpf.plugin INFO : EBPF SOCKET : The range/value (172.16.0.0/12) is inside the range/value (172.16.0.0/12) already inserted, it will be ignored.

2021-09-23 13:13:50: ebpf.plugin INFO : EBPF VFS : thread created with task id 22291

libbpf: elf: skipping unrecognized data section(26) .eh_frame

2021-09-23 13:13:50: ebpf.plugin INFO : EBPF VFS : set name of thread 22291 to EBPF VFS

libbpf: elf: skipping relo section(27) .rel.eh_frame for section(26) .eh_frame

2021-09-23 13:13:50: ebpf.plugin INFO : EBPF FILESYSTEM : thread created with task id 22292

2021-09-23 13:13:50: ebpf.plugin INFO : EBPF FILESYSTEM : set name of thread 22292 to EBPF FILESYSTEM

2021-09-23 13:13:50: ebpf.plugin INFO : EBPF DISK : thread created with task id 22293

2021-09-23 13:13:50: ebpf.plugin INFO : EBPF DISK : set name of thread 22293 to EBPF DISK

2021-09-23 13:13:50: ebpf.plugin INFO : EBPF HARDIRQ : thread created with task id 22296

2021-09-23 13:13:50: ebpf.plugin INFO : EBPF HARDIRQ : set name of thread 22296 to EBPF HARDIRQ

2021-09-23 13:13:50: ebpf.plugin INFO : EBPF SOFTIRQ : thread created with task id 22297

2021-09-23 13:13:50: ebpf.plugin INFO : EBPF SOFTIRQ : set name of thread 22297 to EBPF SOFTIRQ

2021-09-23 13:13:50: ebpf.plugin INFO : EBPF FD : thread created with task id 22295

2021-09-23 13:13:50: ebpf.plugin INFO : EBPF FD : set name of thread 22295 to EBPF FD

2021-09-23 13:13:50: ebpf.plugin INFO : EBPF MOUNT : thread created with task id 22294

libbpf: elf: skipping unrecognized data section(34) .eh_frame

libbpf: elf: skipping unrecognized data section(34) .eh_frame

libbpf: elf: skipping relo section(35) .rel.eh_frame for section(34) .eh_frame

2021-09-23 13:13:50: ebpf.plugin INFO : EBPF MOUNT : set name of thread 22294 to EBPF MOUNT

libbpf: elf: skipping relo section(35) .rel.eh_frame for section(34) .eh_frame

libbpf: elf: skipping unrecognized data section(22) .eh_frame

libbpf: elf: skipping relo section(23) .rel.eh_frame for section(22) .eh_frame

2021-09-23 13:13:50: ebpf.plugin INFO : EBPF OOMKILL : thread created with task id 22298

libbpf: elf: skipping unrecognized data section(22) .eh_frame

libbpf: load bpf program failed: Invalid argument

libbpf: elf: skipping unrecognized data section(16) .eh_frame

2021-09-23 13:13:50: ebpf.plugin INFO : EBPF SOCKET : The range/value (192.168.0.0/16) is inside the range/value (192.168.0.0/16) already inserted, it will be ignored.

libbpf: elf: skipping relo section(23) .rel.eh_frame for section(22) .eh_frame

libbpf: load bpf program failed: Invalid argument

2021-09-23 13:13:50: ebpf.plugin INFO : EBPF SOCKET : The range/value (fc00::/7) is inside the range/value (fc00::/7) already inserted, it will be ignored.

2021-09-23 13:13:50: ebpf.plugin INFO : EBPF SOCKET : The range/value (::1/128) is inside the range/value (::1/128) already inserted, it will be ignored.

2021-09-23 13:13:50: ebpf.plugin INFO : EBPF OOMKILL : set name of thread 22298 to EBPF OOMKILL

libbpf: elf: skipping relo section(17) .rel.eh_frame for section(16) .eh_frame

2021-09-23 13:13:50: ebpf.plugin INFO : EBPF SWAP : thread created with task id 22290

2021-09-23 13:13:50: ebpf.plugin INFO : EBPF SWAP : set name of thread 22290 to EBPF SWAP

libbpf: elf: skipping unrecognized data section(17) .eh_frame

libbpf: elf: skipping relo section(18) .rel.eh_frame for section(17) .eh_frame

libbpf: load bpf program failed: Invalid argument

libbpf: failed to load program 'netdata_sys_write'

libbpf: failed to load object '/usr/libexec/netdata/plugins.d/ebpf.d/pnetdata_ebpf_vfs.4.16.o'

libbpf: failed to load program 'netdata_sys_exit'

libbpf: elf: skipping unrecognized data section(22) .eh_frame

libbpf: elf: skipping unrecognized data section(22) .eh_frame

libbpf: elf: skipping relo section(23) .rel.eh_frame for section(22) .eh_frame

libbpf: failed to load program 'netdata_ext4_file_read_iter'

libbpf: elf: skipping relo section(23) .rel.eh_frame for section(22) .eh_frame

libbpf: failed to load object '/usr/libexec/netdata/plugins.d/ebpf.d/pnetdata_ebpf_ext4.4.16.o'

libbpf: failed to load object '/usr/libexec/netdata/plugins.d/ebpf.d/pnetdata_ebpf_process.4.16.o'

libbpf: elf: skipping unrecognized data section(66) .eh_frame

libbpf: load bpf program failed: Invalid argument

libbpf: failed to load program 'netdata_syscall_sync'

2021-09-23 13:13:50: ebpf.plugin INFO : EBPF SWAP : thread with task id 22290 finishedlibbpf: failed to load object '/usr/libexec/netdata/plugins.d/ebpf.d/pnetdata_ebpf_sync.4.16.o'

2021-09-23 13:13:50: ebpf.plugin ERROR : EBPF SYNC : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ebpf.d/pnetdata_ebpf_sync.4.16.o

(errno 22, Invalid argument)

libbpf: load bpf program failed: Invalid argument

libbpf: elf: skipping relo section(67) .rel.eh_frame for section(66) .eh_frame

libbpf: failed to load program 'netdata_syscall_mount'

2021-09-23 13:13:50: ebpf.plugin ERROR : EBPF FILESYSTEM : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ebpf.d/pnetdata_ebpf_ext4.4.16.o

(errno 22, Invalid argument)

libbpf: failed to load object '/usr/libexec/netdata/plugins.d/ebpf.d/pnetdata_ebpf_mount.4.16.o'

2021-09-23 13:13:50: ebpf.plugin ERROR : EBPF PROCESS : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ebpf.d/pnetdata_ebpf_process.4.16.o

(errno 22, Invalid argument)

2021-09-23 13:13:50: ebpf.plugin ERROR : EBPF VFS : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ebpf.d/pnetdata_ebpf_vfs.4.16.o

(errno 22, Invalid argument)

2021-09-23 13:13:50: ebpf.plugin INFO : EBPF VFS : thread with task id 22291 finished

2021-09-23 13:13:50: ebpf.plugin INFO : EBPF FILESYSTEM : thread with task id 22292 finished

2021-09-23 13:13:50: ebpf.plugin ERROR : EBPF MOUNT : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ebpf.d/pnetdata_ebpf_mount.4.16.o

(errno 22, Invalid argument)

2021-09-23 13:13:50: ebpf.plugin INFO : EBPF MOUNT : thread with task id 22294 finished

2021-09-23 13:13:50: ebpf.plugin INFO : EBPF SYNC : thread with task id 22288 finished

libbpf: elf: skipping unrecognized data section(34) .eh_frame

libbpf: elf: skipping relo section(35) .rel.eh_frame for section(34) .eh_frame

CHART system.softirq_latency '' 'Software IRQ latency' 'milliseconds' 'softirqs' '' 'stacked' 951 1 '' 'ebpf.plugin' 'softirq'

2021-09-23 13:13:50: ebpf.plugin INFO : SOFTIRQ KERNEL : DIMENSION HI HI incremental 1 1

thread created with task id 22305DIMENSION TIMER TIMER incremental 1 1

DIMENSION NET_TX NET_TX incremental 1 1

DIMENSION NET_RX NET_RX incremental 1 1

DIMENSION BLOCK BLOCK incremental 1 1

2021-09-23 13:13:50: ebpf.plugin INFO : SOFTIRQ KERNEL : DIMENSION IRQ_POLL IRQ_POLL incremental 1 1

set name of thread 22305 to SOFTIRQ KERNELDIMENSION TASKLET TASKLET incremental 1 1

DIMENSION SCHED SCHED incremental 1 1

DIMENSION HRTIMER HRTIMER incremental 1 1

DIMENSION RCU RCU incremental 1 1

libbpf: load bpf program failed: Invalid argument

libbpf: failed to load program 'netdata_sys_open'

libbpf: failed to load object '/usr/libexec/netdata/plugins.d/ebpf.d/pnetdata_ebpf_fd.4.16.o'

2021-09-23 13:13:50: ebpf.plugin ERROR : EBPF FD : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ebpf.d/pnetdata_ebpf_fd.4.16.o

(errno 22, Invalid argument)

2021-09-23 13:13:50: ebpf.plugin INFO : EBPF FD : thread with task id 22295 finished

CHART system.hardirq_latency '' 'Hardware IRQ latency' 'milliseconds' 'interrupts' '' 'stacked' 2900 1 '' 'ebpf.plugin' 'hardirq'

DIMENSION apic_thermal apic_thermal incremental 1 1

DIMENSION apic_threshold apic_threshold incremental 1 1

DIMENSION apic_error apic_error incremental 1 1

DIMENSION apic_deferred_error apic_deferred_error incremental 1 1

DIMENSION apic_spurious apic_spurious incremental 1 1

DIMENSION func_call func_call incremental 1 1

DIMENSION func_call_single func_call_single incremental 1 1

DIMENSION reschedule reschedule incremental 1 1

DIMENSION local_timer local_timer incremental 1 1

DIMENSION irq_work irq_work incremental 1 1

DIMENSION x86_platform_ipi x86_platform_ipi incremental 1 1

2021-09-23 13:13:50: ebpf.plugin INFO : HARDIRQ KERNEL : thread created with task id 22308

2021-09-23 13:13:50: ebpf.plugin INFO : HARDIRQ KERNEL : set name of thread 22308 to HARDIRQ KERNEL

2021-09-23 13:13:50: ebpf.plugin INFO : DISK KERNEL : thread created with task id 22310

2021-09-23 13:13:50: ebpf.plugin INFO : DISK KERNEL : set name of thread 22310 to DISK KERNEL

2021-09-23 13:13:50: ebpf.plugin INFO : EBPF DCSTAT : thread with task id 22289 finished

libbpf: load bpf program failed: Invalid argument

libbpf: failed to load program 'netdata_inet_csk_accept'

libbpf: failed to load object '/usr/libexec/netdata/plugins.d/ebpf.d/pnetdata_ebpf_socket.4.16.o'

2021-09-23 13:13:50: ebpf.plugin ERROR : EBPF SOCKET : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ebpf.d/pnetdata_ebpf_socket.4.16.o

(errno 22, Invalid argument)

2021-09-23 13:13:50: ebpf.plugin INFO : EBPF SOCKET : thread with task id 22286 finished

2021-09-23 13:13:52: ebpf.plugin INFO : EBPF PROCESS : thread with task id 22285 finished

I am not updating Ubuntu everyday no. I also am not manually updating netdata each day. I set netdata up 4 weeks ago on these vm’s, got disk latency working for ebpf and didn’t touch it until earlier this week. I went to look at disk latency performance and the entire ebpf menu on the right hand side was gone.

The kernel versions are:
4.15.0-154-generic
4.15.0-156-generic
4.15.0-158-generic

tjk · September 24, 2021, 6:58pm

Anything else you would like me to try @Thiago_Marques_0 ?

tjk · September 26, 2021, 2:22am

OK, I enabled HWE and installed the latest 5.4.0-86-generic #97~18.04.1-Ubuntu and eBPF is working again, so something on the 4.15 kernel was working, and then one of the netdata nightly agent updates broke it.

ilyam8 · October 4, 2021, 7:00am

@Thiago_Marques_0 I have the same problem. Also, when I restart netdata service, the ebpf.plugin keeps hanging until I kill it manually.

My error.log

2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF PROCESS : thread created with task id 29157
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF PROCESS : set name of thread 29157 to EBPF PROCESS
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF SOCKET : thread created with task id 29158
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF SOCKET : set name of thread 29158 to EBPF SOCKET
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF SOCKET : Name resolution is disabled, collector will not parser "hostnames" list.
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF SOCKET : The network value of CIDR 127.0.0.1/8 was updated for 127.0.0.0 .
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF SOCKET : The range/value (127.0.0.1/8) is inside the range/value (127.0.0.1/8) already inserted, it will be ignored.
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF SOCKET : The range/value (10.0.0.0/8) is inside the range/value (10.0.0.0/8) already inserted, it will be ignored.
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF SOCKET : The range/value (172.16.0.0/12) is inside the range/value (172.16.0.0/12) already inserted, it will be ignored.
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF SOCKET : The range/value (192.168.0.0/16) is inside the range/value (192.168.0.0/16) already inserted, it will be ignored.
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF SOCKET : The range/value (fc00::/7) is inside the range/value (fc00::/7) already inserted, it will be ignored.
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF SOCKET : The range/value (::1/128) is inside the range/value (::1/128) already inserted, it will be ignored.
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF CACHESTAT : thread created with task id 29159
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF CACHESTAT : set name of thread 29159 to EBPF CACHESTAT
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF CACHESTAT : thread with task id 29159 finished
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF SYNC : thread created with task id 29160
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF SYNC : set name of thread 29160 to EBPF SYNC
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF DCSTAT : thread created with task id 29161
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF DCSTAT : set name of thread 29161 to EBPF DCSTAT
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF SWAP : thread created with task id 29162
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF SWAP : set name of thread 29162 to EBPF SWAP
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF SWAP : thread with task id 29162 finished
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF FILESYSTEM : thread created with task id 29164
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF FILESYSTEM : set name of thread 29164 to EBPF FILESYSTEM
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF FILESYSTEM : thread with task id 29164 finished
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF MOUNT : thread created with task id 29166
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF MOUNT : set name of thread 29166 to EBPF MOUNT
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF FD : thread created with task id 29167
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF FD : set name of thread 29167 to EBPF FD
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF HARDIRQ : thread created with task id 29168
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF HARDIRQ : set name of thread 29168 to EBPF HARDIRQ
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF SOFTIRQ : thread created with task id 29169
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF SOFTIRQ : set name of thread 29169 to EBPF SOFTIRQ
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF OOMKILL : thread created with task id 29170
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF OOMKILL : set name of thread 29170 to EBPF OOMKILL
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF SHM : thread created with task id 29171
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF SHM : set name of thread 29171 to EBPF SHM
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF DISK : thread created with task id 29165
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF DISK : set name of thread 29165 to EBPF DISK
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF DISK : thread with task id 29165 finished
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF VFS : thread created with task id 29163
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF VFS : set name of thread 29163 to EBPF VFS
2021-10-04 02:58:30: ebpf.plugin ERROR : EBPF FD : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ebpf.d/pnetdata_ebpf_fd.5.4.o
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF FD : thread with task id 29167 finished
2021-10-04 02:58:30: ebpf.plugin ERROR : EBPF MOUNT : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ebpf.d/pnetdata_ebpf_mount.5.4.o
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF MOUNT : thread with task id 29166 finished
2021-10-04 02:58:30: ebpf.plugin ERROR : EBPF SYNC : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ebpf.d/pnetdata_ebpf_sync.5.4.o
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF SYNC : thread with task id 29160 finished
2021-10-04 02:58:30: ebpf.plugin ERROR : EBPF PROCESS : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ebpf.d/pnetdata_ebpf_process.5.4.o
2021-10-04 02:58:30: ebpf.plugin ERROR : EBPF VFS : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ebpf.d/pnetdata_ebpf_vfs.5.4.o
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF VFS : thread with task id 29163 finished
2021-10-04 02:58:30: ebpf.plugin ERROR : EBPF SHM : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ebpf.d/pnetdata_ebpf_shm.5.4.o
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF SHM : thread with task id 29171 finished
2021-10-04 02:58:30: ebpf.plugin INFO  : EBPF DCSTAT : thread with task id 29161 finished
2021-10-04 02:58:31: ebpf.plugin ERROR : EBPF SOCKET : ERROR: loading BPF object file failed /usr/libexec/netdata/plugins.d/ebpf.d/pnetdata_ebpf_socket.5.4.o
2021-10-04 02:58:31: ebpf.plugin INFO  : EBPF SOCKET : thread with task id 29158 finished
2021-10-04 02:58:32: ebpf.plugin INFO  : EBPF PROCESS : thread with task id 29157 finished

System info

Linux deb10-ukraine 4.19.0-17-amd64 #1 SMP Debian 4.19.194-3 (2021-07-18) x86_64 GNU/Linux

I install Netdata with deb package

Thiago_Marques_0 · October 31, 2021, 12:49pm

We had a problem with a libbpf version, but after we update for the latest, the DEB and RPM packages were fixed.

SRE team also merges a PR that also fixed problems for CentOS 7.x, so we are not expecting to have more these problems.

Topic		Replies	Views
Daily Ubuntu 20.04LTS reports _usr_libexec_netdata_plugins.d_ebpf.plugin.0.crash Help agent-health , agent	4	899	December 15, 2020
Netdata crashed during update and eBPF plugin also crashed according to entries in /var/crash Help agent-health , agent	7	914	December 16, 2020
EBPF Errors Help	6	1176	November 3, 2022
eBPF on RHEL7.9 rpm install Help agent	6	846	June 3, 2021
Missing Charts including Applications, Users, etc Help agent	3	1051	June 16, 2022

eBPF not working

Related topics