Node Tab - Custom metrics/add metrics permanent (GPU)

Help
, ,
1

Hey everyone,

Problem/Question

I wanna use a parent-agent system to monitor some nvidia GPU servers. And it’s working.

Then on the nodes tab, I want to see the GPU stats of all the nodes. But I (and bunch of people) don’t want to click at the add metric button, they should be shown as I open the page and therefore this needs to be configured somewhere.

Relevant docs you followed/actions you took to solve the issue

Environment/Browser/Agent’s version etc

v1.41.0

What I expected to happen

An information how I do get this result without using the cloud.

Do I need to go for Custom dashboards | Learn Netdata and Netdata Dashboard as an ugly workaround?

Best!

2

Welcome!

I think the settings are saved here when using Netdata Cloud but not yet able to save them easily via the UI of an agent.

Likewise in NC you could of course also make a custom dashboard.

Neither of these is really possible yet via the agent (we have only recently made cloud and agent UI essentially the same but some stuff like this that requires metadata to be stored somewhere is not yet possible on agent alone).

Are you using Netdata Cloud or the agent dashboard of the parent?

3

Thanks for the fast reply!

I use the node tab of the parent, so no cloud.

but not yet able to save them easily via the UI of an agent.

for me it’s not necessary to save throu the UI by the visitor.

Instead I just want to set some settings in the configuration files (ex. add GPU metrics) and that should then be in the node tab by default for every visitor.

So, there might be a configuration file where I can simple activate those metrics to show?
Or which .html file do I have to edit to add those metrics to the nodes tab overview? Might be in /var/lib/netdata/www/v2?

The least favorable option would be to edit dash-example.html, as I then won’t be in the regular UI for that single page.

4

So I guess it will be the least favourable option: Creating my own dashboard. :confused:

Only thing now is that the API is giving me something like

“nvidia_smi.gpu_gpu-8da5685b-d065-1da7-d233-447d007f5507_fan_speed_perc” instead of “system.swap” so I might have to write my own looping thru these charts (especially as there will be more than one GPU per node).

Is there definitely no easy way to static add the GPUs permanent to the node overview?

5

without using cloud i don’t think so without a little bit of custom work.

actually you might be able to make an old style custom dashboard easy enough on agent - maybe something like this: https://github.com/netdata/community/tree/main/netdata-agent-dashboard/dash-example

Basic idea is use the charting library and make the html you need and then just bundle that with the parent and it would just live at some url on the parent like https://my.netdata.server:19999/dash.html

In fact i think something like that is kinda what you want whereas you may just want to focus on the gpu metrics instead.

fyi - docs for these agent side older custom dashboards are here: Custom dashboards | Learn Netdata

6

Another option could be something like using the netdata pandas library to pull what data you need from the parent agent api and then something like streamlit or dash to make whatever sort of dashboard you want

I’ve done sort of versions of these approaches when building little internal ML MVP apps:

So this could maybe be another approach if you comfortable in Python.