How to see custom variables as a metric in a war room?

I came across this link for exposing metrics of a Golang app: Go applications monitoring with Netdata | Learn Netdata

So I followed these instructions and enabled the default memstats:

app1:
  name : 'myappname'
  url  : 'http://127.0.0.1:8080/debug/vars'
  collect_memstats: true

However, if I go to my war room and try to add one of these application specific metrics, I cannot find anything that might indicate it refers to these enabled memstats. But it is a very long list already, so maybe I miss them because I have no idea what to look for?

Nice you get it working finally

No, there is no need to reboot anything.

Could be this problem netdata.cloud cannot see go.d/mysql · Issue #10116 · netdata/netdata · GitHub (it is not go.d/mysql specific)

I see it is fixed, but not sure it is in production.

@ilyam8

Yes! Now I see them immediately, did not even have to search.

No idea why I did not see these before… Pretty sure I even did a search on the page. Maybe it takes a while to become visible? Maybe it needed a reboot (did that a few times before, but maybe not after solving the parsing problem)?

Anyways, thanks!

If I run it in debug mode then I continues showing values until I terminate it.

as root user

• cd to netdata log dir
• > error.log
• systemctl restart netdata.service
• wait a bit
• grep go_expvar error.log

How to see custom variables as a metric in a war room?

@Fringe did you check agent dashboard? (not via cloud) Open it http://<agent_IP>:19999, go_expvar collector charts should be there

You are absolutely right @fringe. SOrry for dropping the ball on this one

@OdysLam: I think you missed that the go_expvar.conf is now parsed correcly? If I run it in debug mode then I continues showing values until I terminate it. So I guess that part works?

From the debug and your chart, for some reason the go collector has trouble loading the go_expvar.conf file. Can you check that you are using the correct syntax and form as we define it in the documentation (I think that the default configuration file should indicate the correct form).

Thanks for bearing with us! We are getting to the bottom of it!

@ilyam8 I see lots and lots of charts. See the screenshot link that I posted earlier.
It is only that none of these seem to be of my Go app…

I assume this means the job is running correctly.

cannot see any of the memstats of my Go app

Do you see other charts? It is hard to find the problem, sharing your debug output would help

@ilyam8 Ok, I figured this one out

cd /usr/libexec/netdata/plugins.d
sudo -u netdata ./python.d.plugin debug go_expvar

Results in

2020-10-13 14:31:43: python.d INFO: plugin[main] : using python v2
2020-10-13 14:31:43: python.d DEBUG: plugin[main] : looking for 'python.d.conf' in ['/etc/netdata', '/usr/lib/netdata/conf.d']
2020-10-13 14:31:43: python.d DEBUG: plugin[main] : loading '/etc/netdata/python.d.conf'
2020-10-13 14:31:43: python.d DEBUG: plugin[main] : '/etc/netdata/python.d.conf' is loaded
2020-10-13 14:31:43: python.d DEBUG: plugin[main] : looking for 'pythond-jobs-statuses.json' in /var/lib/netdata
2020-10-13 14:31:43: python.d WARNING: plugin[main] : 'pythond-jobs-statuses.json' was not found
2020-10-13 14:31:43: python.d DEBUG: plugin[main] : [go_expvar] looking for 'go_expvar.conf' in ['/etc/netdata/python.d', '/usr/lib/netdata/conf.d/python.d']
2020-10-13 14:31:43: python.d DEBUG: plugin[main] : [go_expvar] loading '/etc/netdata/python.d/go_expvar.conf'
2020-10-13 14:31:43: python.d ERROR: plugin[main] : [go_expvar] error on loading '/etc/netdata/python.d/go_expvar.conf' : ScannerError()
2020-10-13 14:31:43: python.d INFO: plugin[main] : [go_expvar] has no job configs, skipping it
2020-10-13 14:31:43: python.d INFO: plugin[main] : no jobs to run

Great: There must be a problem in my job description!

Solved that

If I now execute the debugging then the log keeps running, showing new data every second and

2020-10-13 14:50:55: python.d DEBUG: go_expvar[app1] : update => [OK] (elapsed time: 2, failed retries in a row: 0)

I assume this means the job is running correctly.

However, still not done. Because I still cannot see any of the memstats of my Go app in either the agent nor the cloud

@OdysLam: this is a screenshot: screenshot-x-x-x-x-19999-2020-10-13-14-15-36 — ImgBB

Hi guys. Checking error.log/using a collector in the debug mode is always a good first step in troubleshooting.

as netdata user

./python.d.plugin debug go_expvar

Hey @fringe,

Before we start debugging the collector, can you send us a screenshot of the sidebar, so we can verify that the charts are not appearing there?

On a side note, even if they do exist, users should be able to find them easily, so that’s definitely some good feedback! Thanks for that

search for a context of interest eg expvar.runtime.goroutines

I exposed the default memstats of my Golang app by including _ "expvar". This results in /debug/vars to return a json with memstats. This works as expected locally on the server commandline with curl.

Within netdata I cannot find any of these custom memory metrics. Not by accessing the agent (:19999) and neither in the cloud. However, no idea if I am looking for the right thing. The long list of data is a rather overwhelming. I hoped memstats.* would be a good start, or expvars.*, but seems not.

I assume the agent that I installed on my VM forwards the data? Is there a way to check if that works correctly?

@Fringe are you seeking for these metrics on the agent or on Cloud?
If on Cloud, you can go to the single node view or add metrics to the nodes table (search for a context of interest eg expvar.runtime.goroutines)