Can't set up and view the example chart

Environment

Nvidia Jetson TX2 machine, OS: Ubuntu (L4T)

Problem/Question

I installed netdata in my jetson board and can view the default charts just fine.
I would like to add my own custom python collectors and view custom charts so I decided to first check the example collector. But I can’t view it.

I have set up:
config file in /etc/netdata/python.d/example.conf
collector in /usr/libexec/netdata/python.d/example.chart.py
And I set example=yes in /etc/netdata/python.d.conf to enable de collector.

Can you help me find what is missing here?
I expected to be able to add the example chart to a dashboard, or to be able to view it in the main netdata page for my node.

Thank you very much in advance.

Hello @Inigo_Arribillaga ,

Welcome to our community!

The steps you described are 100% correct. I remember that either python.d.conf or go.d.conf had to follow a strict pattern for spaces, so I will show parts of my configuration for you compare with yours:

/etc/netdata/python.d.conf

# energid: yes

# this is just an example
example: yes

# exim: yes

/etc/netdata/python.d/example.conf

# AUTO-DETECTION JOBS
# only one of them will run (they have the same name)
python_example:
  v: 1

Please, also run the following command to verify possible errors:

$ grep example /var/log/netdata/error.log | less
2022-03-18 11:21:38: go.d INFO: main[main] looking for 'example.conf' in [/etc/netdata/go.d /usr/lib/netdata/conf.d/go.d]
2022-03-18 11:21:38: go.d INFO: main[main] found '/etc/netdata/go.d/example.conf
2022-03-18 11:21:38: go.d INFO: build[manager] example[go_example] job last state is active/retry, applying recovering settings
2022-03-18 11:21:38: go.d INFO: example[go_example] check success
2022-03-18 11:21:38: go.d INFO: example[go_example] started, data collection interval 1s
2022-03-18 11:21:38: python.d INFO: plugin[main] : [example] built 1 job(s) configs
2022-03-18 11:21:39: netdata INFO  : PLUGINSD[go.d] : INTERNAL CHECK: host 'hades', chart 'example_go_example.random_0' is collected in the same interpolation point: short by 500000 microseconds
2022-03-18 11:21:39: netdata INFO  : PLUGINSD[go.d] : INTERNAL CHECK: host 'hades', chart 'netdata.execution_time_of_example_go_example' is collected in the same interpolation point: short by 500000 microseconds
2022-03-18 11:21:39: python.d INFO: plugin[main] : example[python_example] : check success
2022-03-18 11:21:39: netdata INFO  : PLUGINSD[charts.d] : INTERNAL CHECK: host 'hades', chart 'example.random' is collected in the same interpolation point: short by 500000 microseconds
2022-03-18 11:21:39: netdata INFO  : PLUGINSD[charts.d] : INTERNAL CHECK: host 'hades', chart 'example.random2' is collected in the same interpolation point: short by 500000 microseconds
2022-03-18 11:21:39: netdata INFO  : PLUGINSD[charts.d] : INTERNAL CHECK: host 'hades', chart 'netdata.plugin_chartsd_example' is collected in the same interpolation point: short by 500000 microseconds
2022-03-18 11:21:40: netdata INFO  : PLUGINSD[go.d] : host 'hades', chart 'example_go_example.random_0': discarded data collection time of 1000487 usec, replaced with 500630 usec, reason: 'SYNC CLOCK FLAG'
2022-03-18 11:21:40: netdata INFO  : PLUGINSD[go.d] : host 'hades', chart 'netdata.execution_time_of_example_go_example': discarded data collection time of 1000487 usec, replaced with 500677 usec, reason: 'SYNC CLOCK FLAG'
2022-03-18 11:21:40: netdata INFO  : PLUGINSD[python.d] : INTERNAL CHECK: host 'hades', chart 'example_python_example.random' is collected in the same interpolation point: short by 500000 microseconds
2022-03-18 11:21:40: netdata INFO  : PLUGINSD[python.d] : INTERNAL CHECK: host 'hades', chart 'netdata.runtime_example_python_example' is collected in the same interpolation point: short by 500000 microseconds
2022-03-18 11:21:40: netdata INFO  : PLUGINSD[charts.d] : host 'hades', chart 'example.random': discarded data collection time of 1010000 usec, replaced with 867545 usec, reason: 'SYNC CLOCK FLAG'
2022-03-18 11:21:40: netdata INFO  : PLUGINSD[charts.d] : host 'hades', chart 'example.random2': discarded data collection time of 1010000 usec, replaced with 867582 usec, reason: 'SYNC CLOCK FLAG'
2022-03-18 11:21:40: netdata INFO  : PLUGINSD[charts.d] : host 'hades', chart 'netdata.plugin_chartsd_example': discarded data collection time of 1010000 usec, replaced with 867933 usec, reason: 'SYNC CLOCK FLAG'
2022-03-18 11:21:40: netdata INFO  : PLUGINSD[python.d] : host 'hades', chart 'example_python_example.random': discarded data collection time of 1000991 usec, replaced with 410156 usec, reason: 'SYNC CLOCK FLAG'
2022-03-18 11:21:40: netdata INFO  : PLUGINSD[python.d] : INTERNAL CHECK: host 'hades', chart 'example_python_example.random' is collected in the same interpolation point: short by 89844 microseconds
2022-03-18 11:21:40: netdata INFO  : PLUGINSD[python.d] : host 'hades', chart 'netdata.runtime_example_python_example': discarded data collection time of 1000991 usec, replaced with 410225 usec, reason: 'SYNC CLOCK FLAG'
2022-03-18 11:21:40: netdata INFO  : PLUGINSD[python.d] : INTERNAL CHECK: host 'hades', chart 'netdata.runtime_example_python_example' is collected in the same interpolation point: short by 89775 microseconds

Best regards!

2 Likes

Thanks for the reply, I ran the command to check the error logs as you said and got the following message:
2022-03-18 10:06:49: go.d INFO: main[main] ‘example’ module disabled by default, should be explicitly enabled in the config
2022-03-18 10:06:49: charts.d: INFO: example: is disabled. Add a line with example=force in ‘/etc/netdata/charts.d.conf’ to enable it (or remove the line that disables it).

I thought that as the example code is written in python, I didn’t have to set example=yes in the go config file, but I was wrong. I uncommented the line and can see the chart just fine!

Thank you very much.

OK the chart I’m seeing is the one implemented in Go so I haven’t quite got it yet I think :frowning:

Hello,

We have different examples, and python is one of them.

I installed a VM from scratch, andd I used Arch Linux on it. When I started the VM I got the same result than you:

[root@archlinux vagrant]# grep example /var/log/netdata/error.log 
2022-03-18 20:39:22: go.d INFO: main[main] 'example' module disabled by default, should be explicitly enabled in the config
2022-03-18 20:39:22: charts.d: INFO: example: is disabled. Add a line with example=force in '/etc/netdata/charts.d.conf' to enable it (or remove the line that disables it).

After this I ran the following commands:

$ cd /etc/netdata
$ ./edit-config python.d.conf

And I set example: yes, like you I started my netdata and I did not have the chart, when I ran the following command I got an output that explain everything for me:

[root@archlinux netdata]# grep python.d /var/log/netdata/error.log 
2022-03-18 21:08:15: netdata INFO  : PLUGINSD[python.d] : thread created with task id 2141
2022-03-18 21:08:15: netdata INFO  : PLUGINSD[python.d] : set name of thread 2141 to PLUGINSD[python
2022-03-18 21:08:15: netdata INFO  : PLUGINSD[python.d] : connected to '/usr/libexec/netdata/plugins.d/python.d.plugin' running on pid 2154
/usr/libexec/netdata/plugins.d/python.d.plugin: line 3: which: command not found
/usr/libexec/netdata/plugins.d/python.d.plugin: line 3: which: command not found
/usr/libexec/netdata/plugins.d/python.d.plugin: line 3: which: command not found
2022-03-18 21:08:15: netdata ERROR : PLUGINSD[python.d] : Unknown keyword [ERROR] (errno 9, Bad file descriptor)
2022-03-18 21:08:15: netdata INFO  : PLUGINSD[python.d] : PARSER ended
2022-03-18 21:08:15: netdata ERROR : PLUGINSD[python.d] : '/usr/libexec/netdata/plugins.d/python.d.plugin' (pid 2154) disconnected after 0 successful data collections (ENDs).
2022-03-18 21:08:15: netdata ERROR : PLUGINSD[python.d] : child pid 2154 exited with code 1.
2022-03-18 21:08:15: netdata ERROR : PLUGINSD[python.d] : '/usr/libexec/netdata/plugins.d/python.d.plugin' (pid 2154) exited with error code 1 and haven't collected any data. Disabling it.
2022-03-18 21:08:15: netdata INFO  : PLUGINSD[python.d] : thread with task id 2141 finished

After to see this error, I ran locally pacman -S which and I restart netdata, this time I got the following result:

[root@archlinux netdata]# systemctl start netdata
[root@archlinux netdata]# grep example /var/log/netdata/error.log 
2022-03-18 21:29:21: go.d INFO: main[main] 'example' module disabled by default, should be explicitly enabled in the config
2022-03-18 21:29:22: charts.d: INFO: example: is disabled. Add a line with example=force in '/etc/netdata/charts.d.conf' to enable it (or remove the line that disables it).
2022-03-18 21:29:25: python.d INFO: plugin[main] : [example] built 1 job(s) configs

Please, look for python.d inside your error.log like me, and if you had the same problem, install the package for your distribution.

Best regards,

Thiago

@ilyam8 , on Arch I had whereis, but which was not available. I suggest we use the second when the first is not available.

Thank you for your response Thiago.

After searching for “python” in the error logs, this is what I get:

2022-03-21 07:19:02: netdata INFO : PLUGINSD[python.d] : thread created with task id 12500
2022-03-21 07:19:02: netdata INFO : PLUGINSD[python.d] : set name of thread 12500 to PLUGINSD[python
2022-03-21 07:19:02: netdata INFO : PLUGINSD[python.d] : connected to ‘/usr/libexec/netdata/plugins.d/python.d.plugin’ running on pid 12516
File “/usr/libexec/netdata/plugins.d/python.d.plugin”, line 67, in
2022-03-21 07:19:02: netdata ERROR : PLUGINSD[python.d] : read failed: end of file (errno 9, Bad file descriptor)
2022-03-21 07:19:02: netdata INFO : PLUGINSD[python.d] : PARSER ended
2022-03-21 07:19:02: netdata ERROR : PLUGINSD[python.d] : ‘/usr/libexec/netdata/plugins.d/python.d.plugin’ (pid 12516) disconnected after 0 successful data collections (ENDs).
2022-03-21 07:19:02: netdata ERROR : PLUGINSD[python.d] : child pid 12516 exited with code 1.
2022-03-21 07:19:02: netdata ERROR : PLUGINSD[python.d] : ‘/usr/libexec/netdata/plugins.d/python.d.plugin’ (pid 12516) exited with error code 1 and haven’t collected any data. Disabling it.
2022-03-21 07:19:02: netdata INFO : PLUGINSD[python.d] : thread with task id 12500 finished
2022-03-21 07:19:03: go.d ERROR: prometheus[python_rq_exporter_local] Get “http://127.0.0.1:9726/metrics”: dial tcp 127.0.0.1:9726: connect: connection refused

When debugging the example python collector, I can see the generated values just fine in my terminal. However it seems that netdata keeps trying to display the values taken from the go collector instead of the python one:

prob

This has relationship with line from bases.collection import safe_print. Our coworker @ilyam8 will help us with this package that looks like it is missed.

@Inigo_Arribillaga hi, can you the output from the following command:

/usr/libexec/netdata/plugins.d/python.d.plugin debug trace example nolock

Hello Ilyam, thank you for your reply.

Here is the output:

netdata@dlk00109:/home/iarribillaga$ /usr/libexec/netdata/plugins.d/python.d.plugin debug trace example nolock
2022-03-21 15:34:33: python.d INFO: plugin[main] : using python v3
2022-03-21 15:34:33: python.d DEBUG: plugin[main] : looking for ‘/etc/netdata/python.d.conf’
2022-03-21 15:34:33: python.d DEBUG: plugin[main] : ‘/etc/netdata/python.d.conf’ is loaded
2022-03-21 15:34:33: python.d DEBUG: plugin[main] : looking for ‘/usr/lib/netdata/conf.d/python.d.conf’
2022-03-21 15:34:33: python.d DEBUG: plugin[main] : ‘/usr/lib/netdata/conf.d/python.d.conf’ is loaded
2022-03-21 15:34:33: python.d DEBUG: plugin[main] : looking for ‘pythond-jobs-statuses.json’ in /var/lib/netdata
2022-03-21 15:34:33: python.d DEBUG: plugin[main] : loading ‘/var/lib/netdata/pythond-jobs-statuses.json’
2022-03-21 15:34:33: python.d DEBUG: plugin[main] : ‘/var/lib/netdata/pythond-jobs-statuses.json’ is loaded
2022-03-21 15:34:33: python.d DEBUG: plugin[main] : [example] looking for ‘example.conf’ in [’/etc/netdata/python.d’, ‘/usr/lib/netdata/conf.d/python.d’]
2022-03-21 15:34:33: python.d DEBUG: plugin[main] : [example] loading ‘/usr/lib/netdata/conf.d/python.d/example.conf’
2022-03-21 15:34:33: python.d DEBUG: plugin[main] : [example] ‘/usr/lib/netdata/conf.d/python.d/example.conf’ is loaded
2022-03-21 15:34:33: python.d INFO: plugin[main] : [example] built 1 job(s) configs
2022-03-21 15:34:33: python.d INFO: plugin[main] : example[four_lines] : check success
CHART netdata.runtime_example_Four_Lines ‘’ ‘Execution time for example_Four_Lines’ ‘ms’ ‘python.d’ netdata.pythond_runtime line 145000 1 ‘’ ‘python.d.plugin’ ‘example’
DIMENSION run_time ‘run time’ absolute 1 1

2022-03-21 15:34:33: python.d DEBUG: example[four_lines] : started, update frequency: 1
CHART example_Four_Lines.random ‘’ ‘A random number’ ‘random number’ ‘random’ ‘random’ line 60000 1 ‘’ ‘python.d.plugin’ ‘example’
DIMENSION ‘random1’ ‘random1’ absolute 1 1 ’ ’
DIMENSION ‘random0’ ‘random0’ absolute 1 1 ’ ’
DIMENSION ‘random2’ ‘random2’ absolute 1 1 ’ ’
DIMENSION ‘random3’ ‘random3’ absolute 1 1 ’ ’

BEGIN example_Four_Lines.random 0
SET ‘random1’ = 64
SET ‘random0’ = 70
SET ‘random2’ = 75
SET ‘random3’ = 100
END

I see it works. Check the Agent dashboard, you should see the “Example Charts” section.

Yes, I can see the section. But the chart is not displaying any data:

I have a question, in order to run the debug mode, I had to set the PYTHONPATH variable to point to the netdata python modules folder like this:
export PYTHONPATH=/usr/libexec/netdata/python.d/python_modules.
Maybe I have to put this in some of the config files, like /etc/netdata/netdata.conf? Or in the netdata service file?

When checking the status of netdata service, I can’t see the python plugin active:


Maybe this has something to do too?

You are right, the python.d.plugin is not running.

Let’s check the following

# without specific module (example), let's see if it fails
/usr/libexec/netdata/plugins.d/python.d.plugin debug trace

I don’t think we need that, I suggest to unset.

OK, I’m getting this connection error:

That is expected. Can you copy the whole output (from start till the end) and share it via https://pastebin.com/?

Hello Ilyam, I have solved it finally.

I uninstalled netdata and installed it again but this time from git.
Still, I was getting the same error that I put above in a comment, when I look for ‘python’ in error logs:
File “/usr/libexec/netdata/plugins.d/python.d.plugin”, line 67, in
2022-03-21 07:19:02: netdata ERROR : PLUGINSD[python.d] : read failed: end of file (errno 9, Bad file descriptor

So I went to that file and saw that line 67 was an import from a module in /usr/libexec/netdata/plugins.d/python.d/.

I tried setting the PYTHONPATH to that location in netdata.conf, in [plugin] section and it started working.

Thanks for your attention and help!