So I had an automatic upgrade on a node this morning which upgraded a bunch of stuff including Netdata.
The result was that the web_log collector stopped working and I have a big gap in my charts for today.
The issue was easily resolved via a restart of Netdata but this seems like an area for improvement:
namely, some form of alerting for when we stop getting any metrics which we were already getting (maybe ML can be leveraged here too).
I wonder how we could do this and also just ignore things like ephemeral metrics and stuff like that - maybe there is some heuristics that could be used.
Maybe could also be something in the UI that lets you know and/or gives you some sort of digest or alert.
I know we recently added some indicators on charts for this sort of thing but its not really built out yet and really just a frontend thing for when you are looking at a chart.
I think could be worth a feature request as does make total sense - challenge is in how to do it so it’s not annoying or triggering false alerts by default etc.
@ilyam8 do you know if there is any way at the moment to define an alert that could actually do something like this even if you had to manually create a new one for the special metrics a user might want to monitor?
Wondering - does each metric have a $last_collected_t such that at least a user could make an alert similar to go.d_job_last_collected_secs for any chart or dim?
