Can't set up and view the example chart

Inigo_Arribillaga · March 18, 2022, 8:53am

Environment

Nvidia Jetson TX2 machine, OS: Ubuntu (L4T)

Problem/Question

I installed netdata in my jetson board and can view the default charts just fine.
I would like to add my own custom python collectors and view custom charts so I decided to first check the example collector. But I can’t view it.

I have set up:
config file in /etc/netdata/python.d/example.conf
collector in /usr/libexec/netdata/python.d/example.chart.py
And I set example=yes in /etc/netdata/python.d.conf to enable de collector.

Can you help me find what is missing here?
I expected to be able to add the example chart to a dashboard, or to be able to view it in the main netdata page for my node.

Thank you very much in advance.

Thiago_Marques_0 · March 18, 2022, 11:29am

Hello @Inigo_Arribillaga ,

Welcome to our community!

The steps you described are 100% correct. I remember that either python.d.conf or go.d.conf had to follow a strict pattern for spaces, so I will show parts of my configuration for you compare with yours:

/etc/netdata/python.d.conf

# energid: yes

# this is just an example
example: yes

# exim: yes

/etc/netdata/python.d/example.conf

# AUTO-DETECTION JOBS
# only one of them will run (they have the same name)
python_example:
  v: 1

Please, also run the following command to verify possible errors:

$ grep example /var/log/netdata/error.log | less
2022-03-18 11:21:38: go.d INFO: main[main] looking for 'example.conf' in [/etc/netdata/go.d /usr/lib/netdata/conf.d/go.d]
2022-03-18 11:21:38: go.d INFO: main[main] found '/etc/netdata/go.d/example.conf
2022-03-18 11:21:38: go.d INFO: build[manager] example[go_example] job last state is active/retry, applying recovering settings
2022-03-18 11:21:38: go.d INFO: example[go_example] check success
2022-03-18 11:21:38: go.d INFO: example[go_example] started, data collection interval 1s
2022-03-18 11:21:38: python.d INFO: plugin[main] : [example] built 1 job(s) configs
2022-03-18 11:21:39: netdata INFO  : PLUGINSD[go.d] : INTERNAL CHECK: host 'hades', chart 'example_go_example.random_0' is collected in the same interpolation point: short by 500000 microseconds
2022-03-18 11:21:39: netdata INFO  : PLUGINSD[go.d] : INTERNAL CHECK: host 'hades', chart 'netdata.execution_time_of_example_go_example' is collected in the same interpolation point: short by 500000 microseconds
2022-03-18 11:21:39: python.d INFO: plugin[main] : example[python_example] : check success
2022-03-18 11:21:39: netdata INFO  : PLUGINSD[charts.d] : INTERNAL CHECK: host 'hades', chart 'example.random' is collected in the same interpolation point: short by 500000 microseconds
2022-03-18 11:21:39: netdata INFO  : PLUGINSD[charts.d] : INTERNAL CHECK: host 'hades', chart 'example.random2' is collected in the same interpolation point: short by 500000 microseconds
2022-03-18 11:21:39: netdata INFO  : PLUGINSD[charts.d] : INTERNAL CHECK: host 'hades', chart 'netdata.plugin_chartsd_example' is collected in the same interpolation point: short by 500000 microseconds
2022-03-18 11:21:40: netdata INFO  : PLUGINSD[go.d] : host 'hades', chart 'example_go_example.random_0': discarded data collection time of 1000487 usec, replaced with 500630 usec, reason: 'SYNC CLOCK FLAG'
2022-03-18 11:21:40: netdata INFO  : PLUGINSD[go.d] : host 'hades', chart 'netdata.execution_time_of_example_go_example': discarded data collection time of 1000487 usec, replaced with 500677 usec, reason: 'SYNC CLOCK FLAG'
2022-03-18 11:21:40: netdata INFO  : PLUGINSD[python.d] : INTERNAL CHECK: host 'hades', chart 'example_python_example.random' is collected in the same interpolation point: short by 500000 microseconds
2022-03-18 11:21:40: netdata INFO  : PLUGINSD[python.d] : INTERNAL CHECK: host 'hades', chart 'netdata.runtime_example_python_example' is collected in the same interpolation point: short by 500000 microseconds
2022-03-18 11:21:40: netdata INFO  : PLUGINSD[charts.d] : host 'hades', chart 'example.random': discarded data collection time of 1010000 usec, replaced with 867545 usec, reason: 'SYNC CLOCK FLAG'
2022-03-18 11:21:40: netdata INFO  : PLUGINSD[charts.d] : host 'hades', chart 'example.random2': discarded data collection time of 1010000 usec, replaced with 867582 usec, reason: 'SYNC CLOCK FLAG'
2022-03-18 11:21:40: netdata INFO  : PLUGINSD[charts.d] : host 'hades', chart 'netdata.plugin_chartsd_example': discarded data collection time of 1010000 usec, replaced with 867933 usec, reason: 'SYNC CLOCK FLAG'
2022-03-18 11:21:40: netdata INFO  : PLUGINSD[python.d] : host 'hades', chart 'example_python_example.random': discarded data collection time of 1000991 usec, replaced with 410156 usec, reason: 'SYNC CLOCK FLAG'
2022-03-18 11:21:40: netdata INFO  : PLUGINSD[python.d] : INTERNAL CHECK: host 'hades', chart 'example_python_example.random' is collected in the same interpolation point: short by 89844 microseconds
2022-03-18 11:21:40: netdata INFO  : PLUGINSD[python.d] : host 'hades', chart 'netdata.runtime_example_python_example': discarded data collection time of 1000991 usec, replaced with 410225 usec, reason: 'SYNC CLOCK FLAG'
2022-03-18 11:21:40: netdata INFO  : PLUGINSD[python.d] : INTERNAL CHECK: host 'hades', chart 'netdata.runtime_example_python_example' is collected in the same interpolation point: short by 89775 microseconds

Best regards!

Inigo_Arribillaga · March 18, 2022, 11:51am

Thanks for the reply, I ran the command to check the error logs as you said and got the following message:
2022-03-18 10:06:49: go.d INFO: main[main] ‘example’ module disabled by default, should be explicitly enabled in the config
2022-03-18 10:06:49: charts.d: INFO: example: is disabled. Add a line with example=force in ‘/etc/netdata/charts.d.conf’ to enable it (or remove the line that disables it).

I thought that as the example code is written in python, I didn’t have to set example=yes in the go config file, but I was wrong. I uncommented the line and can see the chart just fine!

Thank you very much.

Inigo_Arribillaga · March 18, 2022, 12:21pm

OK the chart I’m seeing is the one implemented in Go so I haven’t quite got it yet I think

Thiago_Marques_0 · March 18, 2022, 9:32pm

Hello,

We have different examples, and python is one of them.

I installed a VM from scratch, andd I used Arch Linux on it. When I started the VM I got the same result than you:

[root@archlinux vagrant]# grep example /var/log/netdata/error.log 
2022-03-18 20:39:22: go.d INFO: main[main] 'example' module disabled by default, should be explicitly enabled in the config
2022-03-18 20:39:22: charts.d: INFO: example: is disabled. Add a line with example=force in '/etc/netdata/charts.d.conf' to enable it (or remove the line that disables it).

After this I ran the following commands:

$ cd /etc/netdata
$ ./edit-config python.d.conf

And I set example: yes, like you I started my netdata and I did not have the chart, when I ran the following command I got an output that explain everything for me:

[root@archlinux netdata]# grep python.d /var/log/netdata/error.log 
2022-03-18 21:08:15: netdata INFO  : PLUGINSD[python.d] : thread created with task id 2141
2022-03-18 21:08:15: netdata INFO  : PLUGINSD[python.d] : set name of thread 2141 to PLUGINSD[python
2022-03-18 21:08:15: netdata INFO  : PLUGINSD[python.d] : connected to '/usr/libexec/netdata/plugins.d/python.d.plugin' running on pid 2154
/usr/libexec/netdata/plugins.d/python.d.plugin: line 3: which: command not found
/usr/libexec/netdata/plugins.d/python.d.plugin: line 3: which: command not found
/usr/libexec/netdata/plugins.d/python.d.plugin: line 3: which: command not found
2022-03-18 21:08:15: netdata ERROR : PLUGINSD[python.d] : Unknown keyword [ERROR] (errno 9, Bad file descriptor)
2022-03-18 21:08:15: netdata INFO  : PLUGINSD[python.d] : PARSER ended
2022-03-18 21:08:15: netdata ERROR : PLUGINSD[python.d] : '/usr/libexec/netdata/plugins.d/python.d.plugin' (pid 2154) disconnected after 0 successful data collections (ENDs).
2022-03-18 21:08:15: netdata ERROR : PLUGINSD[python.d] : child pid 2154 exited with code 1.
2022-03-18 21:08:15: netdata ERROR : PLUGINSD[python.d] : '/usr/libexec/netdata/plugins.d/python.d.plugin' (pid 2154) exited with error code 1 and haven't collected any data. Disabling it.
2022-03-18 21:08:15: netdata INFO  : PLUGINSD[python.d] : thread with task id 2141 finished

After to see this error, I ran locally pacman -S which and I restart netdata, this time I got the following result:

[root@archlinux netdata]# systemctl start netdata
[root@archlinux netdata]# grep example /var/log/netdata/error.log 
2022-03-18 21:29:21: go.d INFO: main[main] 'example' module disabled by default, should be explicitly enabled in the config
2022-03-18 21:29:22: charts.d: INFO: example: is disabled. Add a line with example=force in '/etc/netdata/charts.d.conf' to enable it (or remove the line that disables it).
2022-03-18 21:29:25: python.d INFO: plugin[main] : [example] built 1 job(s) configs

Please, look for python.d inside your error.log like me, and if you had the same problem, install the package for your distribution.

Best regards,

Thiago

Thiago_Marques_0 · March 18, 2022, 9:32pm

@ilyam8 , on Arch I had whereis, but which was not available. I suggest we use the second when the first is not available.

Inigo_Arribillaga · March 21, 2022, 8:11am

Thank you for your response Thiago.

After searching for “python” in the error logs, this is what I get:

2022-03-21 07:19:02: netdata INFO : PLUGINSD[python.d] : thread created with task id 12500
2022-03-21 07:19:02: netdata INFO : PLUGINSD[python.d] : set name of thread 12500 to PLUGINSD[python
2022-03-21 07:19:02: netdata INFO : PLUGINSD[python.d] : connected to ‘/usr/libexec/netdata/plugins.d/python.d.plugin’ running on pid 12516
File “/usr/libexec/netdata/plugins.d/python.d.plugin”, line 67, in
2022-03-21 07:19:02: netdata ERROR : PLUGINSD[python.d] : read failed: end of file (errno 9, Bad file descriptor)
2022-03-21 07:19:02: netdata INFO : PLUGINSD[python.d] : PARSER ended
2022-03-21 07:19:02: netdata ERROR : PLUGINSD[python.d] : ‘/usr/libexec/netdata/plugins.d/python.d.plugin’ (pid 12516) disconnected after 0 successful data collections (ENDs).
2022-03-21 07:19:02: netdata ERROR : PLUGINSD[python.d] : child pid 12516 exited with code 1.
2022-03-21 07:19:02: netdata ERROR : PLUGINSD[python.d] : ‘/usr/libexec/netdata/plugins.d/python.d.plugin’ (pid 12516) exited with error code 1 and haven’t collected any data. Disabling it.
2022-03-21 07:19:02: netdata INFO : PLUGINSD[python.d] : thread with task id 12500 finished
2022-03-21 07:19:03: go.d ERROR: prometheus[python_rq_exporter_local] Get “http://127.0.0.1:9726/metrics”: dial tcp 127.0.0.1:9726: connect: connection refused

Inigo_Arribillaga · March 21, 2022, 11:55am

When debugging the example python collector, I can see the generated values just fine in my terminal. However it seems that netdata keeps trying to display the values taken from the go collector instead of the python one:

prob

Thiago_Marques_0 · March 21, 2022, 12:43pm

This has relationship with line from bases.collection import safe_print. Our coworker @ilyam8 will help us with this package that looks like it is missed.

ilyam8 · March 21, 2022, 2:04pm

@Inigo_Arribillaga hi, can you the output from the following command:

/usr/libexec/netdata/plugins.d/python.d.plugin debug trace example nolock

Inigo_Arribillaga · March 21, 2022, 3:35pm

Hello Ilyam, thank you for your reply.

Here is the output:

netdata@dlk00109:/home/iarribillaga$ /usr/libexec/netdata/plugins.d/python.d.plugin debug trace example nolock
2022-03-21 15:34:33: python.d INFO: plugin[main] : using python v3
2022-03-21 15:34:33: python.d DEBUG: plugin[main] : looking for ‘/etc/netdata/python.d.conf’
2022-03-21 15:34:33: python.d DEBUG: plugin[main] : ‘/etc/netdata/python.d.conf’ is loaded
2022-03-21 15:34:33: python.d DEBUG: plugin[main] : looking for ‘/usr/lib/netdata/conf.d/python.d.conf’
2022-03-21 15:34:33: python.d DEBUG: plugin[main] : ‘/usr/lib/netdata/conf.d/python.d.conf’ is loaded
2022-03-21 15:34:33: python.d DEBUG: plugin[main] : looking for ‘pythond-jobs-statuses.json’ in /var/lib/netdata
2022-03-21 15:34:33: python.d DEBUG: plugin[main] : loading ‘/var/lib/netdata/pythond-jobs-statuses.json’
2022-03-21 15:34:33: python.d DEBUG: plugin[main] : ‘/var/lib/netdata/pythond-jobs-statuses.json’ is loaded
2022-03-21 15:34:33: python.d DEBUG: plugin[main] : [example] looking for ‘example.conf’ in [‘/etc/netdata/python.d’, ‘/usr/lib/netdata/conf.d/python.d’]
2022-03-21 15:34:33: python.d DEBUG: plugin[main] : [example] loading ‘/usr/lib/netdata/conf.d/python.d/example.conf’
2022-03-21 15:34:33: python.d DEBUG: plugin[main] : [example] ‘/usr/lib/netdata/conf.d/python.d/example.conf’ is loaded
2022-03-21 15:34:33: python.d INFO: plugin[main] : [example] built 1 job(s) configs
2022-03-21 15:34:33: python.d INFO: plugin[main] : example[four_lines] : check success
CHART netdata.runtime_example_Four_Lines ‘’ ‘Execution time for example_Four_Lines’ ‘ms’ ‘python.d’ netdata.pythond_runtime line 145000 1 ‘’ ‘python.d.plugin’ ‘example’
DIMENSION run_time ‘run time’ absolute 1 1

2022-03-21 15:34:33: python.d DEBUG: example[four_lines] : started, update frequency: 1
CHART example_Four_Lines.random ‘’ ‘A random number’ ‘random number’ ‘random’ ‘random’ line 60000 1 ‘’ ‘python.d.plugin’ ‘example’
DIMENSION ‘random1’ ‘random1’ absolute 1 1 ’ ’
DIMENSION ‘random0’ ‘random0’ absolute 1 1 ’ ’
DIMENSION ‘random2’ ‘random2’ absolute 1 1 ’ ’
DIMENSION ‘random3’ ‘random3’ absolute 1 1 ’ ’

BEGIN example_Four_Lines.random 0
SET ‘random1’ = 64
SET ‘random0’ = 70
SET ‘random2’ = 75
SET ‘random3’ = 100
END

ilyam8 · March 21, 2022, 6:45pm

I see it works. Check the Agent dashboard, you should see the “Example Charts” section.

Inigo_Arribillaga · March 22, 2022, 7:46am

Yes, I can see the section. But the chart is not displaying any data:

I have a question, in order to run the debug mode, I had to set the PYTHONPATH variable to point to the netdata python modules folder like this:
export PYTHONPATH=/usr/libexec/netdata/python.d/python_modules.
Maybe I have to put this in some of the config files, like /etc/netdata/netdata.conf? Or in the netdata service file?

Inigo_Arribillaga · March 22, 2022, 8:46am

When checking the status of netdata service, I can’t see the python plugin active:

Maybe this has something to do too?

ilyam8 · March 22, 2022, 10:28am

You are right, the python.d.plugin is not running.

Let’s check the following

# without specific module (example), let's see if it fails
/usr/libexec/netdata/plugins.d/python.d.plugin debug trace

I don’t think we need that, I suggest to unset.

Inigo_Arribillaga · March 22, 2022, 11:07am

OK, I’m getting this connection error:

ilyam8 · March 23, 2022, 4:01pm

That is expected. Can you copy the whole output (from start till the end) and share it via https://pastebin.com/?

Inigo_Arribillaga · March 24, 2022, 8:00am

Hello Ilyam, I have solved it finally.

I uninstalled netdata and installed it again but this time from git.
Still, I was getting the same error that I put above in a comment, when I look for ‘python’ in error logs:
File “/usr/libexec/netdata/plugins.d/python.d.plugin”, line 67, in
2022-03-21 07:19:02: netdata ERROR : PLUGINSD[python.d] : read failed: end of file (errno 9, Bad file descriptor

So I went to that file and saw that line 67 was an import from a module in /usr/libexec/netdata/plugins.d/python.d/.

I tried setting the PYTHONPATH to that location in netdata.conf, in [plugin] section and it started working.

Thanks for your attention and help!

Topic		Replies	Views
No sensor charts showing Help agent	5	1914	December 10, 2020
Custom python.d agent won't start, debug runs fine, example runs fine Help agent , collectors	4	703	May 27, 2022
Charts from oracledb collector not showing Help agent-collector , agent	6	840	July 16, 2021
go.d.plugin not loading modules Help agent	5	1150	April 1, 2023
Writing a custom Python plugin Help agent	12	1718	February 12, 2021

Can't set up and view the example chart

Environment

Problem/Question

Related Topics