Can you alarm on a missing metric?

Help
1

I’m defining my alarms in my nixos configuration, and so I know what services will be running and which alarms to expect.

In particular, I’d like to monitor a certain prometeus metric for each service I define. However, if one of my services isn’t running, the prometeus port isn’t there, and netdata silently ignores it, and the alarm isn’t generated for that service.

Is it possible to force-define an alarm?

I’m assuming each alarm is defined on: a metric, so maybe I can generate dummy alarms on a certain metric that will always exist with only a single instance? And then use calc:to refer to the actual metric I want to check?

2

Hi @wmertens

So, yes, an alarm does need the on chart to exist, in order to be able to define it.

Something like what you describe could work though.

You could define an alert on that specific metric, that will have neither warn or crit entries.

On another alert (indeed must be defined on a different chart) it is possible to reference that first alert and perhaps by checking for NAN you could raise a warning or critical.

Haven’t tried it, but we could check.

3

In NixOS, Netdata alarms are configured based on metrics collected by the Netdata agent. If a service is not running or a specific metric is not available, Netdata may not automatically create an alarm for it. However, you can define custom alarms in the Netdata configuration to address this.

To force-define an alarm, you can use the charts.d section in your Netdata configuration. Here’s an example of how you can define a custom chart with a dummy metric, and then create an alarm based on that metric:

services.netdata = {
enable = true;
config = {
# …

charts.d = {
  dummy_metric = {
    title = "Dummy Metric";
    # Define a dummy metric that will always exist
    dimensions = { instances = { ""; } };
    lines = { dummy_line = { metric = "system.dummy_metric"; } };
  };
};

alarms = {
  # Define an alarm based on the dummy metric
  dummy_alarm = {
    name = "Dummy Alarm";
    message = "This is a dummy alarm";
    chart = "dummy_metric";
    # Specify conditions for the alarm
    on = "system.dummy_metric";
    calc = "!=0";
    value = 0;
  };
};

};
};

You can adapt this approach to your specific use case. Just replace the system.dummy_metric with the actual metric you want to monitor, and adjust the conditions and values accordingly.

1 Like
4

Hi Manolis,

I tried a few things but couldn’t make it work.

So suppose I have the numeric metric prometheus.strato.hello, how can I alert when that doesn’t exist on the system?

And more specifically, suppose I have several instances, how can I alert for each of them?