Missing Charts including Applications, Users, etc

Help
1

Problem/Question

On a fresh Ubuntu 22.04 LTS install the applications, users, etc graphs are missing. I have another Ubuntu 22.04 system using the same version including kernel and this does not have this issue.

Relevant docs you followed/actions you took to solve the issue

I noticed the service has an error message for ebpf plugin which the other system does not have.

# service netdata status
● netdata.service - Real time performance monitoring
     Loaded: loaded (/lib/systemd/system/netdata.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2022-06-14 05:16:14 EDT; 16min ago
   Main PID: 11824 (netdata)
      Tasks: 82 (limit: 154433)
     Memory: 90.6M
        CPU: 16.015s
     CGroup: /system.slice/netdata.service
             ├─11824 /usr/sbin/netdata -D -P /var/run/netdata/netdata.pid
             ├─11826 /usr/sbin/netdata --special-spawn-server
             ├─11969 bash /usr/libexec/netdata/plugins.d/tc-qos-helper.sh 1
             ├─11984 /usr/bin/python3 /usr/libexec/netdata/plugins.d/python.d.plugin 1
             ├─11988 /usr/libexec/netdata/plugins.d/freeipmi.plugin 1
             ├─11990 /usr/libexec/netdata/plugins.d/go.d.plugin 1
             └─11992 /usr/libexec/netdata/plugins.d/nfacct.plugin 1

Jun 14 05:16:14 ubuntu systemd[1]: Started Real time performance monitoring.
Jun 14 05:16:15 ubuntu ebpf.plugin[11987]: Your environment does not have BTF file /sys/kernel/btf//vmlinux. The plugin will work with 'legacy' code.
Jun 14 05:16:15 ubuntu ebpf.plugin[11987]: Name resolution is disabled, collector will not parser "hostnames" list.
Jun 14 05:16:15 ubuntu ebpf.plugin[11987]: The network value of CIDR 127.0.0.1/8 was updated for 127.0.0.0 .
Jun 14 05:16:15 ubuntu ebpf.plugin[11987]: Loaded config file '/etc/netdata/apps_groups.conf'
Jun 14 05:16:15 ubuntu ebpf.plugin[11987]: PROCFILE: Cannot open file '/proc/10644/status'
Jun 14 05:16:15 ubuntu ebpf.plugin[11987]: Cannot open /proc/10644/status


# service netdata status
● netdata.service - Real time performance monitoring
     Loaded: loaded (/lib/systemd/system/netdata.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2022-06-11 06:46:11 EDT; 2 days ago
   Main PID: 1584942 (netdata)
      Tasks: 78 (limit: 77019)
     Memory: 361.2M
        CPU: 3h 4min 28.573s
     CGroup: /system.slice/netdata.service
             ├─1584942 /usr/sbin/netdata -D -P /var/run/netdata/netdata.pid
             ├─1585247 /usr/sbin/netdata --special-spawn-server
             ├─1585421 /usr/libexec/netdata/plugins.d/go.d.plugin 1
             ├─1585425 /usr/bin/python3 /usr/libexec/netdata/plugins.d/python.d.plugin 1
             ├─1585427 /usr/libexec/netdata/plugins.d/apps.plugin 1
             ├─2639726 /usr/libexec/netdata/plugins.d/nfacct.plugin 1
             ├─2640849 bash /usr/libexec/netdata/plugins.d/charts.d.plugin 1
             ├─2747924 /usr/libexec/netdata/plugins.d/freeipmi.plugin 1
             └─3293190 bash /usr/libexec/netdata/plugins.d/tc-qos-helper.sh 1

Jun 11 10:49:12 ubuntu ebpf.plugin[2890002]: Does not have a configuration file inside `/etc/netdata/ebpf.d.conf. It will try to load stock file.
Jun 11 10:49:12 ubuntu ebpf.plugin[2890002]: Your environment does not have BTF file /sys/kernel/btf//vmlinux. The plugin will work with 'legacy' code.
Jun 11 10:49:12 ubuntu ebpf.plugin[2890002]: Name resolution is disabled, collector will not parser "hostnames" list.
Jun 11 10:49:12 ubuntu ebpf.plugin[2890002]: The network value of CIDR 127.0.0.1/8 was updated for 127.0.0.0 .
Jun 11 10:49:12 ubuntu ebpf.plugin[2890002]: Cannot read process groups configuration file '/etc/netdata/apps_groups.conf'. Will try '/usr/lib/netdata/conf.d/>
Jun 11 10:49:23 ubuntu ebpf.plugin[2890756]: Does not have a configuration file inside `/etc/netdata/ebpf.d.conf. It will try to load stock file.
Jun 11 10:49:23 ubuntu ebpf.plugin[2890756]: Your environment does not have BTF file /sys/kernel/btf//vmlinux. The plugin will work with 'legacy' code.
Jun 11 10:49:23 ubuntu ebpf.plugin[2890756]: Name resolution is disabled, collector will not parser "hostnames" list.
Jun 11 10:49:23 ubuntu ebpf.plugin[2890756]: The network value of CIDR 127.0.0.1/8 was updated for 127.0.0.0 .
Jun 11 10:49:23 ubuntu ebpf.plugin[2890756]: Cannot read process groups configuration file '/etc/netdata/apps_groups.conf'. Will try '/usr/lib/netdata/conf.d/

Environment/Browser/Agent’s version etc

Ubuntu 22.04 LTS with XanMod 5.18

# netdata -W buildinfo
Version: netdata v1.35.0-13-nightly

# netdata -W buildinfo
Version: netdata v1.35.0-13-nightly
Configure options:  '--build=x86_64-linux-gnu' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--disable-option-checking' '--disable-silent-rules' '--libdir=${prefix}/lib/x86_64-linux-gnu' '--libexecdir=${prefix}/lib/x86_64-linux-gnu' '--disable-maintainer-mode' '--prefix=/usr' '--sysconfdir=/etc' '--localstatedir=/var' '--libdir=/usr/lib' '--libexecdir=/usr/libexec' '--with-user=netdata' '--with-math' '--with-zlib' '--with-webdir=/var/lib/netdata/www' '--disable-dependency-tracking' 'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -ffile-prefix-map=/usr/src/netdata=. -fstack-protector-strong -Wformat -Werror=format-security' 'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -ffile-prefix-map=/usr/src/netdata=. -fstack-protector-strong -Wformat -Werror=format-security'
Install type: binpkg-deb
    Binary architecture: x86_64
    Packaging distro:
Features:
    dbengine:                   YES
    Native HTTPS:               YES
    Netdata Cloud:              YES
    ACLK Next Generation:       YES
    ACLK-NG New Cloud Protocol: YES
    ACLK Legacy:                NO
    TLS Host Verification:      YES
    Machine Learning:           YES
    Stream Compression:         YES
Libraries:
    protobuf:                YES (system)
    jemalloc:                NO
    JSON-C:                  YES
    libcap:                  NO
    libcrypto:               YES
    libm:                    YES
    tcalloc:                 NO
    zlib:                    YES
Plugins:
    apps:                    YES
    cgroup Network Tracking: YES
    CUPS:                    YES
    EBPF:                    YES
    IPMI:                    YES
    NFACCT:                  YES
    perf:                    YES
    slabinfo:                YES
    Xen:                     NO
    Xen VBD Error Tracking:  NO
Exporters:
    AWS Kinesis:             NO
    GCP PubSub:              NO
    MongoDB:                 NO
    Prometheus Remote Write: YES

What I expected to happen

Graphs to load for Applications.

2

I was able to fix the issue partly by fixing the file permissions up.
chmod 4755 /usr/libexec/netdata/plugins.d/apps.plugin

-rwsr-xr-x 1 root netdata 585616 Jun 13 20:28 apps.plugin

The problem is that now i am noticing lots of the below errors in the logs.

022-06-14 12:39:28: apps.plugin ERROR : MAIN : Cannot process /proc/1/io (command 'systemd') (errno 13, Permission denied)
2022-06-14 12:39:28: apps.plugin ERROR : MAIN : Cannot process /proc/2/io (command 'kthreadd') (errno 13, Permission denied)
2022-06-14 12:39:28: apps.plugin ERROR : MAIN : Cannot process /proc/3/io (command 'rcu_gp') (errno 13, Permission denied)
2022-06-14 12:39:28: apps.plugin ERROR : MAIN : Cannot process /proc/4/io (command 'rcu_par_gp') (errno 13, Permission denied)
3

Hello @steve5518 and welcome in our community. You shouldn’t have to change permissions by yourself. Let me forward your ticket (internally)

4

Hello @steve5518,

Welcome to our community.
Firstly let me do a question to try to replicate the issues, because my VM running Ubuntu 22.04 does not have these issues when I compile netdata from scratch. How did you install your netdata?

About the error “Your environment does not have BTF file /sys/kernel/btf//vmlinux. The plugin will work with ‘legacy’ code.” , this means that on Ubuntu they did not compile the current kernel ( 5.15.0-33-generic #34-Ubuntu) with support to BTF files, so instead to use trampolines, ebpf.plugin is forced to use kprobe.

Considering everything you wrote until now, it looks like that at the end of the installation, the plugins were not set with correct permissions. Both appsand ebpf needs this special permission, because they are touching files that were made by user root.

Best regards!