mem.cachestat_ratio doesn't work after upgrading netdata

Suggested template:

Problem/Question

mem.cachestat_ratio doesn’t work for me anymore. And it’s very good/important metric for me. It’s eBPF plugin

Relevant docs you followed/actions you took to solve the issue

netdata@ns3122599:/usr/libexec/netdata/plugins.d$ ./go.d.plugin -d -m ebpf
DBG godplugin/main.go:62 plugin: name=go.d, version=v2.2.6 component=agent
DBG godplugin/main.go:64 current user: name=netdata, uid=111 component=agent
INF godplugin/main.go:68 env HTTP_PROXY '', HTTPS_PROXY '' component=agent
INF agent/agent.go:159 instance is started component=agent
INF agent/setup.go:23 loading config file component=agent
DBG agent/setup.go:31 looking for 'go.d.conf' in [/etc/netdata /usr/lib/netdata/conf.d] component=agent
INF agent/setup.go:38 found '/usr/lib/netdata/conf.d/go.d.conf component=agent
INF agent/setup.go:45 config successfully loaded component=agent
INF agent/agent.go:163 using config: enabled 'true', default_run 'true', max_procs '0' component=agent
INF agent/setup.go:50 loading modules component=agent
INF agent/setup.go:73 enabled/registered modules: 0/124 component=agent
INF agent/agent.go:176 no modules to run component=agent
netdata@ns3122599:/usr/libexec/netdata/plugins.d$ ./ebpf.plugin 
time=2025-02-21T15:06:13.675+00:00 comm=ebpf.plugin source=collector level=info errno="2, No such file or directory" tid=2667617  msg="Does not have a configuration file inside `/etc/netdata/ebpf.d.conf. It will try to load stock file."
time=2025-02-21T15:06:13.680+00:00 comm=ebpf.plugin source=collector level=info tid=2667617  msg="Name resolution is disabled, collector will not parse \"hostnames\" list."
time=2025-02-21T15:06:13.680+00:00 comm=ebpf.plugin source=collector level=info errno="2, No such file or directory" tid=2667617  msg="Loaded config file '/etc/netdata/apps_groups.conf'"
FUNCTION GLOBAL "network-sockets-tracing" 5 "Detailed information about open sockets." "top" 0x13 100
CHART mount_points.call '' 'Calls to mount and umount syscalls' 'calls/s' 'mount (eBPF)' 'mount_points.call' 'line' 2190 5 '' 'ebpf.plugin' 'mount'
DIMENSION mount mount incremental 1 1
DIMENSION umount umount incremental 1 1
CHART mount_points.error '' 'Errors to mount and umount file systems' 'calls/s' 'mount (eBPF)' 'mount_points.error' 'line' 2191 5 '' 'ebpf.plugin' 'mount'
DIMENSION mount mount incremental 1 1
DIMENSION umount umount incremental 1 1
BEGIN mount_points.call
SET mount = 0
SET umount = 0
END
BEGIN mount_points.error
SET mount = 0
SET umount = 0
END
BEGIN mount_points.call
SET mount = 0
SET umount = 0
END
BEGIN mount_points.error
SET mount = 0
SET umount = 0
END
CHART netdata.ebpf_threads '' 'Threads running.' 'boolean' 'ebpf' '' 'line' 140000 10 '' 'ebpf.plugin' 'main'
DIMENSION process process absolute 1 1
DIMENSION cachestat cachestat absolute 1 1
DIMENSION sync sync absolute 1 1
DIMENSION dc dc absolute 1 1
DIMENSION swap swap absolute 1 1
DIMENSION vfs vfs absolute 1 1
DIMENSION filesystem filesystem absolute 1 1
DIMENSION disk disk absolute 1 1
DIMENSION mount mount absolute 1 1
DIMENSION fd fd absolute 1 1
DIMENSION hardirq hardirq absolute 1 1
DIMENSION softirq softirq absolute 1 1
DIMENSION oomkill oomkill absolute 1 1
DIMENSION shm shm absolute 1 1
DIMENSION mdflush mdflush absolute 1 1
CHART netdata.monitoring_pid '' 'Total number of monitored PIDs' 'pids' 'ebpf' 'netdata.ebpf_pids' 'line' 140001 10 '' 'ebpf.plugin' 'main'
DIMENSION user user absolute 1 1
DIMENSION kernel kernel absolute 1 1
CHART netdata.ebpf_life_time '' 'Time remaining for thread.' 'seconds' 'ebpf' '' 'line' 140002 10 '' 'ebpf.plugin' 'main'
DIMENSION process process absolute 1 1
DIMENSION cachestat cachestat absolute 1 1
DIMENSION sync sync absolute 1 1
DIMENSION dc dc absolute 1 1
DIMENSION swap swap absolute 1 1
DIMENSION vfs vfs absolute 1 1
DIMENSION filesystem filesystem absolute 1 1
DIMENSION disk disk absolute 1 1
DIMENSION mount mount absolute 1 1
DIMENSION fd fd absolute 1 1
DIMENSION hardirq hardirq absolute 1 1
DIMENSION softirq softirq absolute 1 1
DIMENSION oomkill oomkill absolute 1 1
DIMENSION shm shm absolute 1 1
DIMENSION mdflush mdflush absolute 1 1
CHART netdata.ebpf_threads_socket '' 'Threads running.' 'boolean' 'ebpf' '' 'line' 140011 10 '' 'ebpf.plugin' 'main'
DIMENSION socket socket absolute 1 1
CHART netdata.ebpf_life_time_socket '' 'Time remaining for thread.' 'seconds' 'ebpf' '' 'line' 140012 10 '' 'ebpf.plugin' 'main'
DIMENSION socket socket absolute 1 1
CHART netdata.ebpf_load_methods '' 'Load info.' 'methods' 'ebpf' '' 'line' 140003 10 '' 'ebpf.plugin' 'process'
DIMENSION legacy legacy absolute 1 1
DIMENSION co-re co-re absolute 1 1
CHART netdata.ebpf_kernel_memory '' 'Memory allocated for hash tables.' 'bytes' 'ebpf' '' 'line' 140004 10 '' 'ebpf.plugin' 'process'
DIMENSION memory_locked memory_locked absolute 1 1
CHART netdata.ebpf_hash_tables_count '' 'Number of hash tables loaded.' 'hash tables' 'ebpf' '' 'line' 140005 10 '' 'ebpf.plugin' 'process'
DIMENSION hash_table hash_table absolute 1 1
CHART netdata.ebpf_hash_tables_per_core '' 'How threads are loading hash/array tables.' 'threads' 'ebpf' '' 'line' 140006 10 '' 'ebpf.plugin' 'process'
DIMENSION per_core per_core absolute 1 1
DIMENSION unique unique absolute 1 1
CHART netdata.ebpf_hash_tables_global_elements '' 'Controllers inside global table' 'rows' 'ebpf' '' 'line' 140007 10 '' 'ebpf.plugin' 'process'
DIMENSION process process absolute 1 1
DIMENSION socket socket absolute 1 1
DIMENSION cachestat cachestat absolute 1 1
DIMENSION sync sync absolute 1 1
DIMENSION dc dc absolute 1 1
DIMENSION swap swap absolute 1 1
DIMENSION vfs vfs absolute 1 1
DIMENSION filesystem filesystem absolute 1 1
DIMENSION disk disk absolute 1 1
DIMENSION mount mount absolute 1 1
DIMENSION fd fd absolute 1 1
DIMENSION hardirq hardirq absolute 1 1
DIMENSION softirq softirq absolute 1 1
DIMENSION oomkill oomkill absolute 1 1
DIMENSION shm shm absolute 1 1
DIMENSION mdflush mdflush absolute 1 1
CHART netdata.ebpf_hash_tables_insert_pid_elements '' 'Elements inserted into PID table' 'rows' 'ebpf' '' 'line' 140008 10 '' 'ebpf.plugin' 'process'
DIMENSION process process incremental 1 1
DIMENSION socket socket incremental 1 1
DIMENSION cachestat cachestat incremental 1 1
DIMENSION dc dc incremental 1 1
DIMENSION swap swap incremental 1 1
DIMENSION vfs vfs incremental 1 1
DIMENSION fd fd incremental 1 1
DIMENSION oomkill oomkill incremental 1 1
DIMENSION shm shm incremental 1 1
CHART netdata.ebpf_hash_tables_remove_pid_elements '' 'Elements removed from PID table' 'rows' 'ebpf' '' 'line' 140009 10 '' 'ebpf.plugin' 'process'
DIMENSION process process incremental 1 1
DIMENSION socket socket incremental 1 1
DIMENSION cachestat cachestat incremental 1 1
DIMENSION dc dc incremental 1 1
DIMENSION swap swap incremental 1 1
DIMENSION vfs vfs incremental 1 1
DIMENSION fd fd incremental 1 1
DIMENSION oomkill oomkill incremental 1 1
DIMENSION shm shm incremental 1 1
BEGIN netdata.ebpf_threads
SET process = 0
SET cachestat = 0
SET sync = 0
SET dc = 0
SET swap = 0
SET vfs = 0
SET filesystem = 0
SET disk = 0
SET mount = 1
SET fd = 0
SET hardirq = 0
SET softirq = 0
SET oomkill = 0
SET shm = 0
SET mdflush = 0
END
BEGIN netdata.monitoring_pid
SET user = 0
SET kernel = 0
END
BEGIN netdata.ebpf_life_time
SET process = 0
SET cachestat = 0
SET sync = 0
SET dc = 0
SET swap = 0
SET vfs = 0
SET filesystem = 0
SET disk = 0
SET mount = 4294967285
SET fd = 0
SET hardirq = 0
SET softirq = 0
SET oomkill = 0
SET shm = 0
SET mdflush = 0
END
BEGIN netdata.ebpf_load_methods
SET legacy = 0
SET co-re = 1
END
BEGIN netdata.ebpf_kernel_memory
SET memory_locked = 864
END
BEGIN netdata.ebpf_hash_tables_count
SET hash_table = 1
END
BEGIN netdata.ebpf_hash_tables_per_core
SET per_core = 1
SET unique = 0
END
BEGIN netdata.ebpf_hash_tables_global_elements
SET process = 0
SET socket = 0
SET cachestat = 0
SET sync = 0
SET dc = 0
SET swap = 0
SET vfs = 0
SET filesystem = 0
SET disk = 0
SET mount = 6
SET fd = 0
SET hardirq = 0
SET softirq = 0
SET oomkill = 0
SET shm = 0
SET mdflush = 0
END
BEGIN netdata.ebpf_hash_tables_insert_pid_elements
SET process = 0
SET socket = 0
SET cachestat = 0
SET dc = 0
SET swap = 0
SET vfs = 0
SET fd = 0
SET oomkill = 0
SET shm = 0
END
BEGIN netdata.ebpf_hash_tables_remove_pid_elements
SET process = 0
SET socket = 0
SET cachestat = 0
SET dc = 0
SET swap = 0
SET vfs = 0
SET fd = 0
SET oomkill = 0
SET shm = 0
END
BEGIN netdata.ebpf_threads_socket
SET socket = 0
END
BEGIN netdata.ebpf_life_time_socket
SET socket = 0
END
BEGIN mount_points.call
SET mount = 0
SET umount = 0
END
BEGIN mount_points.error
SET mount = 0
SET umount = 0
END
BEGIN mount_points.call
SET mount = 0
SET umount = 0
END
BEGIN mount_points.error
SET mount = 0
SET umount = 0
END

Environment/Browser/Agent’s version etc

Node ID

6761e296-9cc4-4006-b686-881dfeba3450

Connection

Status:

Live

Netdata Agent:v2.2.6

What I expected to happen

mem.cachestat_ratio to be returned in netdata

Example of how it works in other older node.

cc @ilyam8 @hugo
Maybe you have any suggestions here?

I am currently reworking the eBPF plugin (ebpf.plugin: rework memory by thiagoftsm · Pull Request #19844 · netdata/netdata · GitHub) to optimize its memory usage.

After seeing your message, I remembered that cachestat was one of the threads I had to modify multiple times due to kernel updates. I’ve tested it on kernel 6.12.18, and it is working as expected.

To help me recreate the issue and address it in the current PR, could you let me know which distribution and kernel version you are using?

Best regards,

Thiago

Thank you for getting back to me @Thiago_Marques_0 with such detailed response! I really appreciate it.

Type:
Bare Metal
O/S version:Ubuntu, 24.04.2 LTS (Noble Numbat)
Architecture:x86_64

Kernel:Linux, 6.8.0-53-generic

Let me know if I can give you more logs or any information that would help you to understand what’s wrong.

For reference: Another nodes
0b32dae8-51b0-43b5-9316-89ff6824e6d0
b34ec721-baf2-465a-862e-d0ccc0b1f7ba

It works. But it’s using really old netdata + old kernel

System info

Netdata Agent:
v1.44.3

Type:

Bare Metal

O/S version:Ubuntu, 20.04.6 LTS (Focal Fossa)
Ubuntu, 22.04.5 LTS (Jammy Jellyfish)

Architecture:x86_64

Kernel:Linux, 5.4.0-196-generic
Linux, 5.15.0-130-generic