Hmm - strange i created a centos machine and installed netdata from source, installed the python packages as per docs, enabled the collector and restarted netdata and it seems to work for me.
In below message the “training complete” line in the logs indicates it looks like its working.
Then if i go to the dashboard i see the initial cpu spike for that training run, then some smallish window while its waiting to build up enough data to get the lagged values to start doing the predictions, during with you will see some error messages like below in the log.
2020-12-07 11:53:14: python.d INFO: anomalies[local] : training complete in 0.24 seconds (runs_counter=1, model=pca, train_n_secs=14400, models=21, n_fit_success=21, n_fit_fails=0, after=1607327594, before=1607341994).
2020-12-07 11:53:45: python.d ERROR: anomalies[local] : update() unhandled exception: index -1 is out of bounds for axis 0 with size 0
2020-12-07 11:53:51: python.d ERROR: anomalies[local] : update() unhandled exception: index -1 is out of bounds for axis 0 with size 0
2020-12-07 11:53:57: python.d ERROR: anomalies[local] : update() unhandled exception: index -1 is out of bounds for axis 0 with size 0
2020-12-07 11:54:03: python.d ERROR: anomalies[local] : update() unhandled exception: index -1 is out of bounds for axis 0 with size 0
2020-12-07 11:54:12: python.d ERROR: anomalies[local] : update() unhandled exception: index -1 is out of bounds for axis 0 with size 0
2020-12-07 11:54:21: python.d ERROR: anomalies[local] : update() unhandled exception: index -1 is out of bounds for axis 0 with size 0
2020-12-07 11:54:30: python.d ERROR: anomalies[local] : update() unhandled exception: index -1 is out of bounds for axis 0 with size 0
2020-12-07 11:54:39: python.d ERROR: anomalies[local] : update() unhandled exception: index -1 is out of bounds for axis 0 with size 0
But then you should see some cpu being used for the prediction steps:
And the chart itself available once you refresh the page.
I’m not sure what might be going wrong for you. Only hint i can think of is that the python error message itself looks like its not liking the python f-strings which were only available from python 3.6+ iirc.
I’m the ML guy so not really an expert on any systems stuff like this but happy to try help you debug it as best we can and ask other in Netdata who might be more knowledgeable.
I do recall one of our developers hit a similar error message here: Anomalies collector by andrewm4894 · Pull Request #10060 · netdata/netdata · GitHub
But it went away once he reinstalled from the branch and/or rebooted the machine. He said the error went away but still not quite sure where it came from or what was driving it.
Let me know if you are still getting it and if you can find anymore info on it.
One thing to try would be to just run the collector in debug mode and see if that gives anymore info or detail in the error message:
# become netdata user
sudo su -s /bin/bash netdata
# run collector in debug using `nolock` option if netdata is already running the collector itself.
/usr/libexec/netdata/plugins.d/python.d.plugin anomalies debug trace nolock