go.d SNMP collector either not collecting or not publishing

Help
,
1

Environment: Debian 11
Netdata: v1.38.0 from netdata repo

I have been trying to integrate SNMP monitoring of a CyberPower PDU81002 based on the MIB definition provided for this device. I have successfully used snmpwalk and other mib browsers to correctly identify the OID values I am interested in.

Since Debian 11 does not yet package the go.d plugin, and the node.d.plugin is now deprecated, I have downloaded the latest netdata debian package from the netdata repo, although I’ll point out that everything I’m about to explain was also problematic using the node plugin.

This device has 8 outlets for which power utilization can be measured in both amps and watts. I ultimately want to monitor all 8 outlets as well as some other statistics (on or off, etc), but I am starting with one outlet and these two oids. My go.d/snmp.conf is as follows:

jobs:
  - name: cps
    hostname: "10.10.10.240"
    community: public
    update_every: 1
    max_request_size: 150
    options: 
      version: 2
    charts:
      - id: "outlet_status_load"
        units: "amps"
        type: "area"
        dimensions:
          - name: "cabinet"
            oid: "1.3.6.1.4.1.3808.1.1.6.6.2.4.1.6.8"
            algorithm: "incremental"
            multiplier: 1
            divisor: 1
      - id: "outlet_status_active_power"
        units: "watts"
        type: "area"
        dimensions:
          - name: "cabinet"
            oid: "1.3.6.1.4.1.3808.1.1.6.6.2.4.1.7.8"
            algorithm: "incremental"
            multiplier: 1
            divisor: 1

When I run /usr/libexec/netdata/plugins.d/go.d.plugin -d I can see it successfully read the two oids correctly at every interval:

BEGIN 'snmp_cps.outlet_status_load' 999555
SET 'cabinet' = 8
END

BEGIN 'snmp_cps.outlet_status_active_power' 999555
SET 'cabinet' = 50
END

which repeats every “tick” interval. (full log attached).

I’ve turned off every other metric in netdata.conf (also attached) and written a custom dashboard to just expose these charts (cps.html, attached)

I get occasional peaks on the amp chart and very isolated peaks on the watt chart, but neither values are correct, or consistent. The amperage load on the outlet being measured is consistently 8 (measured in tenth-amps) and the power in watts is consistently 50 (measured in watts).

What I want to see is a flat line at 8 and 50 so long as the load/power doesn’t fluctuate, and if it does fluctuate, I’d like the graph to show this. I do not want a scissor peak between a value and 0, and I do not want the wrong values! I’ve attached a screen shot of what my custom dashboard shows.

If additional log snippets or configuration is useful, please ask.

Thanks,
-brmiller

go.d.plugin-debug.txt 
[ DEBUG ] main[main] main.go:113 plugin: name=go.d, version=v0.50.0
[ DEBUG ] main[main] main.go:115 current user: name=root, uid=0
[ INFO  ] main[main] agent.go:136 instance is started
[ INFO  ] main[main] setup.go:42 loading config file
[ INFO  ] main[main] setup.go:50 looking for 'go.d.conf' in [/etc/netdata /usr/lib/netdata/conf.d]
[ INFO  ] main[main] setup.go:57 found '/etc/netdata/go.d.conf
[ INFO  ] main[main] setup.go:64 config successfully loaded
[ INFO  ] main[main] agent.go:140 using config: enabled 'true', default_run 'true', max_procs '0'
[ INFO  ] main[main] setup.go:69 loading modules
[ INFO  ] main[main] setup.go:83 'openvpn_status_log' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'ping' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'zookeeper' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'proxysql' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'traefik' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'whoisquery' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'coredns' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'freeradius' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'lighttpd' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'mongodb' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'nginxvts' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'pgbouncer' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'pihole' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'dnsmasq' module disabled in the config file
[ INFO  ] main[main] setup.go:79 'example' module disabled by default, should be explicitly enabled in the config
[ INFO  ] main[main] setup.go:83 'filecheck' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'haproxy' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'nginx' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'vsphere' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'web_log' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'couchdb' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'wmi' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'cassandra' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'chrony' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'dns_query' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'docker_engine' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'powerdns_recursor' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'postgres' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'vernemq' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'wireguard' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'cockroachdb' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'couchbase' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'logstash' module disabled in the config file
[ INFO  ] main[main] setup.go:79 'systemdunits' module disabled by default, should be explicitly enabled in the config
[ INFO  ] main[main] setup.go:83 'tengine' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'consul' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'dnsdist' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'phpfpm' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'redis' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'vcsa' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'x509check' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'docker' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'lighttpd2' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'nginxplus' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'powerdns' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'pulsar' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'hdfs' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'unbound' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'logind' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'mysql' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'supervisord' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'nvme' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'portcheck' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'prometheus' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'dnsmasq_dhcp' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'dockerhub' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'elasticsearch' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'fluentd' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'isc_dhcpd' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'apache' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'geth' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'scaleio' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'phpdaemon' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'pika' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'solr' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'httpcheck' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'k8s_kubeproxy' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'ntpd' module disabled in the config file
[ INFO  ] main[main] setup.go:79 'nvidia_smi' module disabled by default, should be explicitly enabled in the config
[ INFO  ] main[main] setup.go:79 'openvpn' module disabled by default, should be explicitly enabled in the config
[ INFO  ] main[main] setup.go:83 'squidlog' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'rabbitmq' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'springboot2' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'activemq' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'bind' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'energid' module disabled in the config file
[ INFO  ] main[main] setup.go:83 'k8s_kubelet' module disabled in the config file
[ INFO  ] main[main] setup.go:79 'k8s_state' module disabled by default, should be explicitly enabled in the config
[ INFO  ] main[main] setup.go:88 enabled/registered modules: 1/79
[ INFO  ] main[main] setup.go:93 building discovery config
[ INFO  ] main[main] setup.go:123 looking for 'snmp.conf' in [/etc/netdata/go.d /usr/lib/netdata/conf.d/go.d]
[ INFO  ] main[main] setup.go:139 found '/etc/netdata/go.d/snmp.conf
[ INFO  ] main[main] setup.go:144 dummy/read/watch paths: 0/1/0
[ INFO  ] discovery[manager] manager.go:92 registered discoverers: [file discovery: [file reader]]
[ INFO  ] discovery[manager] manager.go:97 instance is started
[ INFO  ] discovery[file manager] discovery.go:73 instance is started
[ INFO  ] run[manager] run.go:32 instance is started
[ INFO  ] build[manager] build.go:107 instance is started
[ INFO  ] discovery[file reader] read.go:41 instance is started
[ INFO  ] discovery[file reader] read.go:42 instance is stopped
[ DEBUG ] build[manager] build.go:154 received config group ('/etc/netdata/go.d/snmp.conf'): 1 jobs (added: 1, removed: 0)
[ DEBUG ] build[manager] build.go:303 building snmp[cps] job, config: map[__provider__:file reader __source__:/etc/netdata/go.d/snmp.conf autodetection_retry:0 charts:[map[dimensions:[map[algorithm:incremental divisor:1 multiplier:1 name:cabinet oid:1.3.6.1.4.1.3808.1.1.6.6.2.4.1.6.8]] id:outlet_status_load type:area units:amps] map[dimensions:[map[algorithm:incremental divisor:1 multiplier:1 name:cabinet oid:1.3.6.1.4.1.3808.1.1.6.6.2.4.1.7.8]] id:outlet_status_active_power type:area units:watts]] community:public hostname:172.16.10.240 max_request_size:150 module:snmp name:cps options:map[version:2] priority:70000 update_every:1]
[ INFO  ] snmp[cps] snmp.go:117 hostname=172.16.10.240,port=161,snmp_version=2c,community=public
[ INFO  ] snmp[cps] job.go:194 check success
[ INFO  ] snmp[cps] job.go:214 started, data collection interval 1s
[ DEBUG ] run[manager] run.go:43 tick 0
CHART 'netdata.execution_time_of_snmp_cps' '' 'Execution time' 'ms' 'go.d' 'netdata.go_plugin_execution_time' 'line' '145000' '1' '' 'go.d' 'snmp'
CLABEL '_collect_job' 'cps' '1'
CLABEL_COMMIT
DIMENSION 'time' '' 'absolute' '1' '1' ''

CHART 'snmp_cps.outlet_status_load' '' 'Untitled chart' 'amps' '' 'snmp.outlet_status_load' 'area' '70000' '1' '' 'go.d' 'snmp'
CLABEL '_collect_job' 'cps' '1'
CLABEL_COMMIT
DIMENSION 'cabinet' 'cabinet' 'incremental' '1' '1' ''

BEGIN 'snmp_cps.outlet_status_load'
SET 'cabinet' = 8
END

CHART 'snmp_cps.outlet_status_active_power' '' 'Untitled chart' 'watts' '' 'snmp.outlet_status_active_power' 'area' '70000' '1' '' 'go.d' 'snmp'
CLABEL '_collect_job' 'cps' '1'
CLABEL_COMMIT
DIMENSION 'cabinet' 'cabinet' 'incremental' '1' '1' ''

BEGIN 'snmp_cps.outlet_status_active_power'
SET 'cabinet' = 50
END

BEGIN 'netdata.execution_time_of_snmp_cps'
SET 'time' = 1
END

[ DEBUG ] run[manager] run.go:43 tick 1
BEGIN 'snmp_cps.outlet_status_load' 999555
SET 'cabinet' = 8
END

BEGIN 'snmp_cps.outlet_status_active_power' 999555
SET 'cabinet' = 50
END

BEGIN 'netdata.execution_time_of_snmp_cps' 999555
SET 'time' = 1
END

[ DEBUG ] run[manager] run.go:43 tick 2
BEGIN 'snmp_cps.outlet_status_load' 1000290
SET 'cabinet' = 8
END

BEGIN 'snmp_cps.outlet_status_active_power' 1000290
SET 'cabinet' = 50
END

BEGIN 'netdata.execution_time_of_snmp_cps' 1000290
SET 'time' = 0
END

^C[ INFO  ] main[main] agent.go:104 received interrupt signal (2). Terminating...
[ INFO  ] run[manager] run.go:33 instance is stopped
[ INFO  ] discovery[manager] manager.go:98 instance is stopped
[ INFO  ] build[manager] build.go:108 instance is stopped
[ INFO  ] discovery[file manager] discovery.go:74 instance is stopped
[ INFO  ] snmp[cps] job.go:215 stopped
[ INFO  ] main[main] agent.go:137 instance is stopped
netdata.conf 
[global]
    run as user = netdata

[db]
    # default storage size - increase for longer data retention
    dbengine page cache size MB = 32
    dbengine multihost disk space MB = 256

[web]
    #web files owner = root
    #web files group = root
    # Netdata is not designed to be exposed to potentially hostile
    # networks. See https://github.com/netdata/netdata/issues/164
    bind to = 10.106.249.194

[plugins]
    timex = no
    idlejitter = no
    netdata monitoring = no
    tc = no
    diskspace = no
    proc = no
    cgroups = no
    enable running new plugins = no
    check for new plugins every = 60
    slabinfo = no
    ebpf = no
    charts.d = no
    go.d = yes
    python.d = no
    nfacct = no
    ioping = no
    perf = no
    apps = no
    statsd = no
cps.html 
<!DOCTYPE html>
<html lang="en">
<head>
  <title>Your dashboard</title>

  <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
  <meta charset="utf-8">
  <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <meta name="apple-mobile-web-app-capable" content="yes">
  <meta name="apple-mobile-web-app-status-bar-style" content="black-translucent">

  <!-- here we will add dashboard.js -->
  <script type="text/javascript" src="http://10.106.249.194:19999/dashboard.js"></script>


</head>
<body>

<!-- here we will add charts -->
<div data-netdata="snmp_cps.outlet_status_load"></div>
<div data-netdata="snmp_cps.outlet_status_active_power"></div>

</body>
</html>

2023-04-04-103819_412x374_scrot

2

I don’t see the X axis on the charts screenshots, but it definitely doesn’t look right. It’s extremely unlikely for the chart to NOT show exactly what was collected. Aggregations (average by default) and interpolation does happen, but I can’t think of any possibility for the charts you’re showing, if the Amps are indeed steady at 8 and 50. You can double check just in case though, by taking the actual /api/v1/data query issued to Netdata and putting it on your browser.

E.g. from this custom chart, with the console open, I just need to double-click on the URL that appears in the network tab

image
image1838×737 85.4 KB

I get the page http://localhost:19999/api/v1/data?chart=netdata.server_cpu&format=json&points=1272&group=average&gtime=0&options=ms|flip|jsonwrap|nonzero&after=1680882050&before=1680883250&_=1680883009085 which returns the data points that the chart visualizes.

To eliminate the possibility of aggregation/interpolation, I can set the number of points to be the same as the interval I request.
e.g. points=60&after=-60 will return the last 60 seconds, with one data point per second.

I expect you will find that indeed there are wrong values stored in Netdata for every second. As I said, I can’t think of a scenario where the plugin in debug mode would show the correct numbers and store in Netdata wrong numbers. Can you leave the debug mode running while at the same time doing queries with the API for the last 60 seconds? If they don’t match (as it seems they don’t), maybe @ilyam8 can help further.

3

Oh, the link to the documentation of the /api/v1/data call is Netdata API

4

Hi, @brmiller. I think you need to change

algorithm: "incremental"

to

algorithm: "absolute"

incremental should be used for monotonically increasing values only.

  • incremental: Netdata calculates per second rate and stores it in the database.
  • absolute: Netdata stores the value as is without any processing.
5

Netdata provides pre-built deb packages. See Install Netdata using native DEB/RPM packages.

6

This was the issue. As I copied from the SNMP device collector README I did not realize the author’s measurement of packets transferred was of a different value interpretation than my “absolute” PDU readings. Thanks for pointing this out @ilyam8. After making the change, I get

2023-04-07-112448_584x370_scrot

@Christopher_Akritid1, thanks for the API spec information - I have used this subsequently to further refine my custom chart HTML when I am not getting the right results.

Lastly, I am using the Netdata-built packages since Debian 11 doesn’t package the latest - thanks!

-brmiller

7

since Debian 11 doesn’t package the latest

Do you mean the latest go.d.plugin or what? Because we have both stable and latest Netdata repositories. The latest contains the most recent changes from the Netdata’s master branch.

8

The Debian project itself does not package the go.d.plugin at all. Bullseye still packages an older netdata and the deprecated nodejs plugin which is what I started with. When I couldn’t get it to work I noticed that the Netdata project itself has moved to go, but because Debian doesn’t package it [yet], I switched to the Netdata repo.

-brmiller

9

FYI I added a line to the documentation of the collector with Update README.md by cakrit · Pull Request #1150 · netdata/go.d.plugin · GitHub