Netdata Community

Python sensors not visible

For some reason the “new” Python based sensors are not working for me at all. I’m probably missing something silly but I can’t figure it out.

I have tried:

  • a “blank” config for python.d/sensors.conf
  • a config with the specific chips enabled
  • enabling pretty much all settings such as update_every: 1 and such

I’m not using a custom dashboard, everything is pretty much stock and I have even tried completely wiping my netdata install.

What works:
sensors command properly shows the sensors as expected:

# sensors
iwlwifi_1-virtual-0
Adapter: Virtual device
temp1:        +37.0°C

it8628-isa-0a60
Adapter: ISA adapter
in0:           1.98 V  (min =  +0.00 V, max =  +3.06 V)
in1:           3.06 V  (min =  +0.00 V, max =  +3.06 V)  ALARM
in2:           1.02 V  (min =  +0.00 V, max =  +3.06 V)
+5V:           1.01 V  (min =  +0.00 V, max =  +6.12 V)
in4:           1.97 V  (min =  +0.00 V, max =  +3.06 V)
in5:           1.30 V  (min =  +0.00 V, max =  +3.06 V)
in6:           1.57 V  (min =  +0.00 V, max =  +3.06 V)
3VSB:          3.67 V  (min =  +0.00 V, max =  +6.12 V)
Vbat:          3.41 V
fan1:         875 RPM  (min =   10 RPM)
fan2:        1030 RPM  (min =   10 RPM)
fan3:         772 RPM  (min =   10 RPM)
fan4:           0 RPM  (min =    0 RPM)
fan5:          -1 RPM  (min =    0 RPM)
temp1:        +46.0°C  (low  = +127.0°C, high = +127.0°C)  sensor = thermistor
temp2:        +50.0°C  (low  = +127.0°C, high = +127.0°C)  sensor = thermistor
temp3:        +40.0°C  (low  = +127.0°C, high = +127.0°C)  sensor = thermistor
temp4:       -110.0°C
temp5:        +60.0°C
temp6:         +0.0°C
intrusion0:  ALARM

nouveau-pci-0b00
Adapter: PCI adapter
fan1:           0 RPM
temp1:        +44.0°C  (high = +95.0°C, hyst =  +3.0°C)
                       (crit = +105.0°C, hyst =  +5.0°C)
                       (emerg = +135.0°C, hyst =  +5.0°C)

acpitz-acpi-0
Adapter: ACPI interface
temp1:        +16.8°C  (crit = +20.8°C)

k10temp-pci-00c3
Adapter: PCI adapter
Tctl:         +52.2°C
Tdie:         +52.2°C
Tccd1:        +36.5°C
Tccd2:        +37.0°C

it8628-isa-0a40
Adapter: ISA adapter
CPU Vcore:       1.43 V  (min =  +0.00 V, max =  +3.06 V)
+3.3V:           2.06 V  (min =  +0.00 V, max =  +3.06 V)
+12V:            2.05 V  (min =  +0.00 V, max =  +3.06 V)
+5V:             2.00 V  (min =  +0.00 V, max =  +3.06 V)
CPU Vcore SOC:   1.01 V  (min =  +0.00 V, max =  +3.06 V)
CPU Vddp:      912.00 mV (min =  +0.00 V, max =  +3.06 V)
DRAM A/B:        1.38 V  (min =  +0.00 V, max =  +3.06 V)
3VSB:            3.34 V  (min =  +0.00 V, max =  +6.12 V)
Battery:         3.19 V
CPU_FAN:       4753 RPM  (min =   10 RPM)
SYS_FAN1:       922 RPM  (min =   10 RPM)
SYS_FAN2:       906 RPM  (min =   10 RPM)
SYS_FAN3:      1220 RPM  (min =   10 RPM)
CPU_OPT:          0 RPM  (min =    0 RPM)
System 1:       +36.0°C  (low  = +127.0°C, high = +127.0°C)  sensor = thermistor
Chipset:        +33.0°C  (low  = +127.0°C, high = +127.0°C)  sensor = thermistor
CPU Socket:     +52.0°C  (low  = +127.0°C, high = +127.0°C)  sensor = disabled
PCIEX16:        +46.0°C
VRM MOS:        +42.0°C
VSOC MOS:       +49.0°C
intrusion0:    ALARM

nvme-pci-0400
Adapter: PCI adapter
Composite:    +45.9°C  (low  = -273.1°C, high = +84.8°C)
                       (crit = +84.8°C)
Sensor 1:     +45.9°C  (low  = -273.1°C, high = +65261.8°C)
Sensor 2:     +46.9°C  (low  = -273.1°C, high = +65261.8°C)

Debugging the plugin also shows output:

# /usr/libexec/netdata/plugins.d/python.d.plugin sensors debug trace
2021-09-14 23:58:15: python.d INFO: plugin[main] : using python v3
2021-09-14 23:58:15: python.d DEBUG: plugin[main] : looking for '/etc/netdata/python.d.conf'
2021-09-14 23:58:15: python.d INFO: plugin[main] : '/etc/netdata/python.d.conf' was not found
2021-09-14 23:58:15: python.d DEBUG: plugin[main] : looking for '/usr/lib/netdata/conf.d/python.d.conf'
2021-09-14 23:58:15: python.d DEBUG: plugin[main] : '/usr/lib/netdata/conf.d/python.d.conf' is loaded
2021-09-14 23:58:15: python.d DEBUG: plugin[main] : looking for 'pythond-jobs-statuses.json' in /var/lib/netdata
2021-09-14 23:58:15: python.d DEBUG: plugin[main] : loading '/var/lib/netdata/pythond-jobs-statuses.json'
2021-09-14 23:58:15: python.d DEBUG: plugin[main] : '/var/lib/netdata/pythond-jobs-statuses.json' is loaded
2021-09-14 23:58:15: python.d DEBUG: plugin[main] : [sensors] looking for 'sensors.conf' in ['/etc/netdata/python.d', '/usr/lib/netdata/conf.d/python.d']
2021-09-14 23:58:15: python.d DEBUG: plugin[main] : [sensors] loading '/etc/netdata/python.d/sensors.conf'
2021-09-14 23:58:15: python.d DEBUG: plugin[main] : [sensors] '/etc/netdata/python.d/sensors.conf' is loaded
2021-09-14 23:58:15: python.d INFO: plugin[main] : [sensors] built 1 job(s) configs
2021-09-14 23:58:17: python.d INFO: plugin[main] : sensors[sensors] : check success
CHART netdata.runtime_sensors '' 'Execution time for sensors' 'ms' 'python.d' netdata.pythond_runtime line 145000 10 '' 'python.d.plugin' 'sensors'
DIMENSION run_time 'run time' absolute 1 1

2021-09-14 23:58:17: python.d DEBUG: sensors[sensors] : started, update frequency: 10
CHART sensors.iwlwifi_1-virtual-0_temperature '' 'iwlwifi_1-virtual-0 temperature' 'Celsius' 'temperature' 'sensors.temperature' line 60000 10 '' 'python.d.plugin' 'sensors'
DIMENSION 'iwlwifi_1-virtual-0_temp1' 'temp1' absolute 1 1000 ' '

BEGIN sensors.iwlwifi_1-virtual-0_temperature 0
SET 'iwlwifi_1-virtual-0_temp1' = 35000
END

CHART sensors.it8628-isa-0a60_temperature '' 'it8628-isa-0a60 temperature' 'Celsius' 'temperature' 'sensors.temperature' line 60001 10 '' 'python.d.plugin' 'sensors'
DIMENSION 'it8628-isa-0a60_temp1' 'temp1' absolute 1 1000 ' '
DIMENSION 'it8628-isa-0a60_temp2' 'temp2' absolute 1 1000 ' '
DIMENSION 'it8628-isa-0a60_temp3' 'temp3' absolute 1 1000 ' '
DIMENSION 'it8628-isa-0a60_temp4' 'temp4' absolute 1 1000 ' '
DIMENSION 'it8628-isa-0a60_temp5' 'temp5' absolute 1 1000 ' '
DIMENSION 'it8628-isa-0a60_temp6' 'temp6' absolute 1 1000 ' '

BEGIN sensors.it8628-isa-0a60_temperature 0
SET 'it8628-isa-0a60_temp1' = 46000
SET 'it8628-isa-0a60_temp2' = 51000
SET 'it8628-isa-0a60_temp3' = 40000
SET 'it8628-isa-0a60_temp4' = -110000
SET 'it8628-isa-0a60_temp5' = 63000
SET 'it8628-isa-0a60_temp6' = -1000
END

CHART sensors.nouveau-pci-0b00_temperature '' 'nouveau-pci-0b00 temperature' 'Celsius' 'temperature' 'sensors.temperature' line 60002 10 '' 'python.d.plugin' 'sensors'
DIMENSION 'nouveau-pci-0b00_temp1' 'temp1' absolute 1 1000 ' '

BEGIN sensors.nouveau-pci-0b00_temperature 0
SET 'nouveau-pci-0b00_temp1' = 44000
END

CHART sensors.acpitz-acpi-0_temperature '' 'acpitz-acpi-0 temperature' 'Celsius' 'temperature' 'sensors.temperature' line 60003 10 '' 'python.d.plugin' 'sensors'
DIMENSION 'acpitz-acpi-0_temp1' 'temp1' absolute 1 1000 ' '

BEGIN sensors.acpitz-acpi-0_temperature 0
SET 'acpitz-acpi-0_temp1' = 16800
END

CHART sensors.k10temp-pci-00c3_temperature '' 'k10temp-pci-00c3 temperature' 'Celsius' 'temperature' 'sensors.temperature' line 60004 10 '' 'python.d.plugin' 'sensors'
DIMENSION 'k10temp-pci-00c3_temp1' 'Tctl' absolute 1 1000 ' '
DIMENSION 'k10temp-pci-00c3_temp2' 'Tdie' absolute 1 1000 ' '
DIMENSION 'k10temp-pci-00c3_temp3' 'Tccd1' absolute 1 1000 ' '
DIMENSION 'k10temp-pci-00c3_temp4' 'Tccd2' absolute 1 1000 ' '

BEGIN sensors.k10temp-pci-00c3_temperature 0
SET 'k10temp-pci-00c3_temp1' = 54375
SET 'k10temp-pci-00c3_temp2' = 54375
SET 'k10temp-pci-00c3_temp3' = 36250
SET 'k10temp-pci-00c3_temp4' = 37000
END

CHART sensors.it8628-isa-0a40_temperature '' 'it8628-isa-0a40 temperature' 'Celsius' 'temperature' 'sensors.temperature' line 60005 10 '' 'python.d.plugin' 'sensors'
DIMENSION 'it8628-isa-0a40_temp1' 'System 1' absolute 1 1000 ' '
DIMENSION 'it8628-isa-0a40_temp2' 'Chipset' absolute 1 1000 ' '
DIMENSION 'it8628-isa-0a40_temp3' 'CPU Socket' absolute 1 1000 ' '
DIMENSION 'it8628-isa-0a40_temp4' 'PCIEX16' absolute 1 1000 ' '
DIMENSION 'it8628-isa-0a40_temp5' 'VRM MOS' absolute 1 1000 ' '
DIMENSION 'it8628-isa-0a40_temp6' 'VSOC MOS' absolute 1 1000 ' '

BEGIN sensors.it8628-isa-0a40_temperature 0
SET 'it8628-isa-0a40_temp1' = 36000
SET 'it8628-isa-0a40_temp2' = 33000
SET 'it8628-isa-0a40_temp3' = 54000
SET 'it8628-isa-0a40_temp4' = 46000
SET 'it8628-isa-0a40_temp5' = 42000
SET 'it8628-isa-0a40_temp6' = 49000
END

CHART sensors.nvme-pci-0400_temperature '' 'nvme-pci-0400 temperature' 'Celsius' 'temperature' 'sensors.temperature' line 60006 10 '' 'python.d.plugin' 'sensors'
DIMENSION 'nvme-pci-0400_temp1' 'Composite' absolute 1 1000 ' '
DIMENSION 'nvme-pci-0400_temp2' 'Sensor 1' absolute 1 1000 ' '
DIMENSION 'nvme-pci-0400_temp3' 'Sensor 2' absolute 1 1000 ' '

BEGIN sensors.nvme-pci-0400_temperature 0
SET 'nvme-pci-0400_temp1' = 45850
SET 'nvme-pci-0400_temp2' = 45850
SET 'nvme-pci-0400_temp3' = 46850
END

CHART sensors.it8628-isa-0a60_fan '' 'it8628-isa-0a60 fans speed' 'Rotations/min' 'fans' 'sensors.fan' line 60007 10 '' 'python.d.plugin' 'sensors'
DIMENSION 'it8628-isa-0a60_fan1' 'fan1' absolute 1 1000 ' '
DIMENSION 'it8628-isa-0a60_fan2' 'fan2' absolute 1 1000 ' '
DIMENSION 'it8628-isa-0a60_fan3' 'fan3' absolute 1 1000 ' '
DIMENSION 'it8628-isa-0a60_fan4' 'fan4' absolute 1 1000 ' '
DIMENSION 'it8628-isa-0a60_fan5' 'fan5' absolute 1 1000 ' '

BEGIN sensors.it8628-isa-0a60_fan 0
SET 'it8628-isa-0a60_fan1' = 873000
SET 'it8628-isa-0a60_fan2' = 1035000
SET 'it8628-isa-0a60_fan3' = 772000
SET 'it8628-isa-0a60_fan4' = 0
END

CHART sensors.nouveau-pci-0b00_fan '' 'nouveau-pci-0b00 fans speed' 'Rotations/min' 'fans' 'sensors.fan' line 60008 10 '' 'python.d.plugin' 'sensors'
DIMENSION 'nouveau-pci-0b00_fan1' 'fan1' absolute 1 1000 ' '

BEGIN sensors.nouveau-pci-0b00_fan 0
SET 'nouveau-pci-0b00_fan1' = 0
END

CHART sensors.it8628-isa-0a40_fan '' 'it8628-isa-0a40 fans speed' 'Rotations/min' 'fans' 'sensors.fan' line 60009 10 '' 'python.d.plugin' 'sensors'
DIMENSION 'it8628-isa-0a40_fan1' 'CPU_FAN' absolute 1 1000 ' '
DIMENSION 'it8628-isa-0a40_fan2' 'SYS_FAN1' absolute 1 1000 ' '
DIMENSION 'it8628-isa-0a40_fan3' 'SYS_FAN2' absolute 1 1000 ' '
DIMENSION 'it8628-isa-0a40_fan4' 'SYS_FAN3' absolute 1 1000 ' '
DIMENSION 'it8628-isa-0a40_fan5' 'CPU_OPT' absolute 1 1000 ' '

BEGIN sensors.it8628-isa-0a40_fan 0
SET 'it8628-isa-0a40_fan1' = 4753000
SET 'it8628-isa-0a40_fan2' = 922000
SET 'it8628-isa-0a40_fan3' = 902000
SET 'it8628-isa-0a40_fan4' = 1222000
SET 'it8628-isa-0a40_fan5' = 0
END

CHART sensors.it8628-isa-0a60_voltage '' 'it8628-isa-0a60 voltage' 'Volts' 'voltage' 'sensors.voltage' line 60010 10 '' 'python.d.plugin' 'sensors'
DIMENSION 'it8628-isa-0a60_in0' 'in0' absolute 1 1000 ' '
DIMENSION 'it8628-isa-0a60_in1' 'in1' absolute 1 1000 ' '
DIMENSION 'it8628-isa-0a60_in2' 'in2' absolute 1 1000 ' '
DIMENSION 'it8628-isa-0a60_in3' '+5V' absolute 1 1000 ' '
DIMENSION 'it8628-isa-0a60_in4' 'in4' absolute 1 1000 ' '
DIMENSION 'it8628-isa-0a60_in5' 'in5' absolute 1 1000 ' '
DIMENSION 'it8628-isa-0a60_in6' 'in6' absolute 1 1000 ' '
DIMENSION 'it8628-isa-0a60_in7' '3VSB' absolute 1 1000 ' '
DIMENSION 'it8628-isa-0a60_in8' 'Vbat' absolute 1 1000 ' '

BEGIN sensors.it8628-isa-0a60_voltage 0
SET 'it8628-isa-0a60_in0' = 1968
SET 'it8628-isa-0a60_in1' = 3060
SET 'it8628-isa-0a60_in2' = 1020
SET 'it8628-isa-0a60_in3' = 1416
SET 'it8628-isa-0a60_in4' = 1968
SET 'it8628-isa-0a60_in5' = 1296
SET 'it8628-isa-0a60_in6' = 1560
SET 'it8628-isa-0a60_in7' = 3672
SET 'it8628-isa-0a60_in8' = 3408
END

CHART sensors.it8628-isa-0a40_voltage '' 'it8628-isa-0a40 voltage' 'Volts' 'voltage' 'sensors.voltage' line 60011 10 '' 'python.d.plugin' 'sensors'
DIMENSION 'it8628-isa-0a40_in0' 'CPU Vcore' absolute 1 1000 ' '
DIMENSION 'it8628-isa-0a40_in1' '+3.3V' absolute 1 1000 ' '
DIMENSION 'it8628-isa-0a40_in2' '+12V' absolute 1 1000 ' '
DIMENSION 'it8628-isa-0a40_in3' '+5V' absolute 1 1000 ' '
DIMENSION 'it8628-isa-0a40_in4' 'CPU Vcore SOC' absolute 1 1000 ' '
DIMENSION 'it8628-isa-0a40_in5' 'CPU Vddp' absolute 1 1000 ' '
DIMENSION 'it8628-isa-0a40_in6' 'DRAM A/B' absolute 1 1000 ' '
DIMENSION 'it8628-isa-0a40_in7' '3VSB' absolute 1 1000 ' '
DIMENSION 'it8628-isa-0a40_in8' 'Battery' absolute 1 1000 ' '

BEGIN sensors.it8628-isa-0a40_voltage 0
SET 'it8628-isa-0a40_in0' = 1248
SET 'it8628-isa-0a40_in1' = 2064
SET 'it8628-isa-0a40_in2' = 2052
SET 'it8628-isa-0a40_in3' = 2004
SET 'it8628-isa-0a40_in4' = 1008
SET 'it8628-isa-0a40_in5' = 912
SET 'it8628-isa-0a40_in6' = 1380
SET 'it8628-isa-0a40_in7' = 3336
SET 'it8628-isa-0a40_in8' = 3192
END

BEGIN netdata.runtime_sensors 0
SET run_time = 258
END

2021-09-14 23:58:25: python.d DEBUG: sensors[sensors] : update => [OK] (elapsed time: 258, failed retries in a row: 0)
^C2021-09-14 23:58:26: python.d INFO: plugin[main] : exiting from main...

Environment

Installed using the kickstart script and I have attempted a reinstall as well.
Version: v1.31.0-245-nightly
OS: Ubuntu 20.04 LTS

The netdata logs don’t contain any information about sensors. i.e. grep -i sensors /var/log/netdata/*.log returns nothing

Problem/Question

How can I make the sensors show in netdata?

What I expected to happen

Visible sensors in the dashboard
I have also just added a nut (for UPS) chart and I had no issues with that.

Hi, @wolph. Thanks for the detailed post and using formatting :+1:

Let’s do the following:

  • do ps faxu | grep "[n]etdata" and show the output.
  • restart netdata.service; wait for 2-3 seconds; do grep sensors /var/log/netdata/error.log and show the output.

Hah, that’s a cool grep trick to prevent a “grep -v grep” :stuck_out_tongue:

# ps faxu | grep '[n]etdata'
netdata     8873  1.8  0.3 535460 251408 ?       Ssl  Sep15  26:41 /usr/sbin/netdata -P /var/run/netdata/netdata.pid -D
netdata     8893  0.0  0.0  31840  4576 ?        Sl   Sep15   0:00  \_ /usr/sbin/netdata --special-spawn-server
netdata     9174  2.8  0.0  68532 47064 ?        S    Sep15  40:51  \_ /usr/libexec/netdata/plugins.d/apps.plugin 1
netdata     9178  0.0  0.0 725688 15508 ?        Sl   Sep15   0:21  \_ /usr/libexec/netdata/plugins.d/go.d.plugin 1
root        9179  4.1  0.3 1024524 202024 ?      Sl   Sep15  58:57  \_ /usr/libexec/netdata/plugins.d/ebpf.plugin 1
netdata  3181869  0.0  0.0   4244  3192 ?        S    01:37   0:01  \_ bash /usr/libexec/netdata/plugins.d/tc-qos-helper.sh 1

The grep only shows the charts.d sensors, which is disabled as expected. But I don’t see the python.d sensors anywhere.

# grep sensors *
error.log:2021-09-16 02:15:06: charts.d: INFO: sensors: is disabled. Add a line with sensors=force in '/etc/netdata/charts.d.conf' to enable it (or remove the line that disables it).

Ok, for some reason python.d.plugin is not running :man_shrugging:

Let’s check grep python.d error.log

Here are the logs. I’m seeing some errors, but also success.

# grep 'python\.d' error.log
2021-09-17 01:05:25: netdata INFO  : PLUGINSD[python.d] : thread created with task id 3158879
2021-09-17 01:05:25: netdata INFO  : PLUGINSD[python.d] : set name of thread 3158879 to PLUGINSD[python
2021-09-17 01:05:25: netdata INFO  : PLUGINSD[python.d] : connected to '/usr/libexec/netdata/plugins.d/python.d.plugin' running on pid 3158884
/usr/libexec/netdata/plugins.d/python.d.plugin: line 18: /usr/bin/python: Operation not permitted
2021-09-17 01:05:25: netdata INFO  : WEB_SERVER[static1] : /usr/libexec/netdata/plugins.d/python.d.plugin: line 18: /usr/bin/python: Success
2021-09-17 01:05:25: netdata ERROR : PLUGINSD[python.d] : read failed: end of file (errno 9, Bad file descriptor)
2021-09-17 01:05:25: netdata INFO  : PLUGINSD[python.d] : PARSER ended
2021-09-17 01:05:25: netdata ERROR : PLUGINSD[python.d] : '/usr/libexec/netdata/plugins.d/python.d.plugin' (pid 3158884) disconnected after 0 successful data collections (ENDs).
2021-09-17 01:05:25: netdata ERROR : PLUGINSD[python.d] : child pid 3158884 exited with code 126.
2021-09-17 01:05:25: netdata ERROR : PLUGINSD[python.d] : '/usr/libexec/netdata/plugins.d/python.d.plugin' (pid 3158884) exited with error code 126 and haven't collected any data. Disabling it.
2021-09-17 01:05:25: netdata INFO  : PLUGINSD[python.d] : thread with task id 3158879 finished

It seems that this line is not working fully: exec "$pybinary" "$0" "${filtered[@]}"
But I don’t see why… when I manually execute the command it’s working fine. Also when logged in as the netdata user. The /usr/bin/python symlink points to the system python 3.8 install

Does it work when you run it manually as “netdata” user?

Yes, that works perfectly.

So I guess we can narrow it down to something with how systemd is running it and/or environmental settings. I’ve patched the python.d file to log the environmental settings, and these are the differences:

23d22
< PIDFILE=/run/netdata/netdata.pid
39,40d37
< INVOCATION_ID=0a503e558f6d47d8baf87195a2b9b5a6
< RUNTIME_DIRECTORY=/run/netdata
48a46
> TERM=screen-256color
51a50
> LIBVIRT_DEFAULT_URI=qemu:///system
63d61
< JOURNAL_STREAM=8:29849974
65c63,64
< PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin
---
> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin
> MAIL=/var/mail/netdata

The right one is working, the left one isn’t. I don’t see anything that jumps out honestly…
I’ve also compared the limits between them and nothing useful in that area either:

7c7
< max locked memory       (kbytes, -l) 64
---
> max locked memory       (kbytes, -l) 65536

Is there anything else different from when systemd launches it?