How to see custom variables as a metric in a war room?



  • I came across this link for exposing metrics of a Golang app: https://learn.netdata.cloud/docs/agent/collectors/python.d.plugin/go_expvar

    So I followed these instructions and enabled the default memstats:

    app1:
      name : 'myappname'
      url  : 'http://127.0.0.1:8080/debug/vars'
      collect_memstats: true
    

    However, if I go to my war room and try to add one of these application specific metrics, I cannot find anything that might indicate it refers to these enabled memstats. But it is a very long list already, so maybe I miss them because I have no idea what to look for?


  • Staff

    @Fringe are you seeking for these metrics on the agent or on Cloud?
    If on Cloud, you can go to the single node view or add metrics to the nodes table (search for a context of interest eg expvar.runtime.goroutines)



  • search for a context of interest eg expvar.runtime.goroutines

    I exposed the default memstats of my Golang app by including _ "expvar". This results in <url>/debug/vars to return a json with memstats. This works as expected locally on the server commandline with curl.

    Within netdata I cannot find any of these custom memory metrics. Not by accessing the agent (<url>:19999) and neither in the cloud. However, no idea if I am looking for the right thing. The long list of data is a rather overwhelming. I hoped memstats.* would be a good start, or expvars.*, but seems not.

    I assume the agent that I installed on my VM forwards the data? Is there a way to check if that works correctly?


  • Staff

    Hey @fringe,

    Before we start debugging the collector, can you send us a screenshot of the sidebar, so we can verify that the charts are not appearing there?

    On a side note, even if they do exist, users should be able to find them easily, so that’s definitely some good feedback! Thanks for that πŸ™‚


  • Staff

    Hi guys. Checking error.log/using a collector in the debug mode is always a good first step in troubleshooting.

    as netdata user

    ./python.d.plugin debug go_expvar



  • @OdysLam: this is a screenshot: https://ibb.co/jLhPyWQ



  • @ilyam8 Ok, I figured this one out

    cd /usr/libexec/netdata/plugins.d
    sudo -u netdata ./python.d.plugin debug go_expvar
    

    Results in

    2020-10-13 14:31:43: python.d INFO: plugin[main] : using python v2
    2020-10-13 14:31:43: python.d DEBUG: plugin[main] : looking for 'python.d.conf' in ['/etc/netdata', '/usr/lib/netdata/conf.d']
    2020-10-13 14:31:43: python.d DEBUG: plugin[main] : loading '/etc/netdata/python.d.conf'
    2020-10-13 14:31:43: python.d DEBUG: plugin[main] : '/etc/netdata/python.d.conf' is loaded
    2020-10-13 14:31:43: python.d DEBUG: plugin[main] : looking for 'pythond-jobs-statuses.json' in /var/lib/netdata
    2020-10-13 14:31:43: python.d WARNING: plugin[main] : 'pythond-jobs-statuses.json' was not found
    2020-10-13 14:31:43: python.d DEBUG: plugin[main] : [go_expvar] looking for 'go_expvar.conf' in ['/etc/netdata/python.d', '/usr/lib/netdata/conf.d/python.d']
    2020-10-13 14:31:43: python.d DEBUG: plugin[main] : [go_expvar] loading '/etc/netdata/python.d/go_expvar.conf'
    2020-10-13 14:31:43: python.d ERROR: plugin[main] : [go_expvar] error on loading '/etc/netdata/python.d/go_expvar.conf' : ScannerError()
    2020-10-13 14:31:43: python.d INFO: plugin[main] : [go_expvar] has no job configs, skipping it
    2020-10-13 14:31:43: python.d INFO: plugin[main] : no jobs to run
    

    Great: There must be a problem in my job description! πŸ™‚

    Solved that πŸ‘

    If I now execute the debugging then the log keeps running, showing new data every second and

    2020-10-13 14:50:55: python.d DEBUG: go_expvar[app1] : update => [OK] (elapsed time: 2, failed retries in a row: 0)
    

    I assume this means the job is running correctly.

    However, still not done. Because I still cannot see any of the memstats of my Go app in either the agent nor the cloud 😞


  • Staff

    I assume this means the job is running correctly.

    πŸŽ‰

    cannot see any of the memstats of my Go app

    Do you see other charts? It is hard to find the problem, sharing your debug output would help



  • @ilyam8 I see lots and lots of charts. See the screenshot link that I posted earlier.
    It is only that none of these seem to be of my Go app…


  • Staff

    From the debug and your chart, for some reason the go collector has trouble loading the go_expvar.conf file. Can you check that you are using the correct syntax and form as we define it in the documentation (I think that the default configuration file should indicate the correct form).

    Thanks for bearing with us! We are getting to the bottom of it!



  • @OdysLam: I think you missed that the go_expvar.conf is now parsed correcly? If I run it in debug mode then I continues showing values until I terminate it. So I guess that part works?


Log in to reply