# Trigger an alarm when a chart never reaches zero over the last 24 hours **URL:** \ **Category:** Help\ **Tags:** agent, alerts\ **Created:** [July 16, 2023, 12:45pm UTC](https://community.netdata.cloud/t/trigger-an-alarm-when-a-chart-never-reaches-zero-over-the-last-24-hours/4403 "2023-07-16T12:45:13Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![nodiscc](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/nodiscc/32/2982_2.png) [@nodiscc](https://community.netdata.cloud/u/nodiscc)\ **Post date:** [July 16, 2023, 12:45pm UTC](https://community.netdata.cloud/t/trigger-an-alarm-when-a-chart-never-reaches-zero-over-the-last-24-hours/4403/1 "2023-07-16T12:45:13Z")
## Problem/Question Hi, I want to trigger an alarm when a particular chart on my netdata instance never reaches zero over the last 24 hours. (For more context, the chart is generated by [this plugin](https://github.com/nodiscc/netdata-apt)) I have configured `/etc/netdata/health.d/apt.conf` in the following way: ```auto alarm: apt_upgradable on: apt.upgradable lookup: min -1d of upgradable every: 60s warn: $this > 0 units: packages info: packages with available upgrades to: sysadmin ``` However, this alarm stays active, even though the `apt.upgradable` chart value has been 0 for several hours. My goal is for this alarm to be active **only** when the chart value has constantly been \> 0 for the last day. ## Relevant docs you followed/actions you took to solve the issue - Double-checked that my lookup expression was correct on [Configure Health Alerts | Learn Netdata](https://learn.netdata.cloud/docs/alerting/health-configuration-reference#expressions) (it doesn’t mention the `min` operator) - Tried to find other example uses of the `min` operator in stock netdata health configuration - Tried changing the health configuration file in a few ways (`warn: $this != nan AND $this > 0`…), without results Here is the chart over the last 26 hours. ![apt](https://canada1.discourse-cdn.com/flex029/uploads/netdata2/original/2X/6/6bc848a1db17cf11fadc5651dea6fd053c258031.png) ## Environment/Browser/Agent’s version etc - Debian 11/12 - Netdata agent 1.40.1 ## What I expected to happen In these conditions I don’t expect the alarm to be active. ---
**Author:** ![Manolis\_Vasilakis](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/manolis_vasilakis/32/820_2.png) [@Manolis\_Vasilakis](https://community.netdata.cloud/u/Manolis_Vasilakis)\ **Post date:** [July 17, 2023, 4:23pm UTC](https://community.netdata.cloud/t/trigger-an-alarm-when-a-chart-never-reaches-zero-over-the-last-24-hours/4403/2 "2023-07-17T16:23:24Z")
Hi @nodiscc You could perhaps use `sum`. When the sum is 0 over the last day, the alert will be clear. If there is a package to be upgraded it will raise to warning. However, clearing after a warning would take a whole day without any packages to update, so not sure if you’d like that. I would maybe suggest to also lower the day to maybe an hour. I’ll try to think of a better way. ---
**Author:** ![nodiscc](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/nodiscc/32/2982_2.png) [@nodiscc](https://community.netdata.cloud/u/nodiscc)\ **Post date:** [July 17, 2023, 4:35pm UTC](https://community.netdata.cloud/t/trigger-an-alarm-when-a-chart-never-reaches-zero-over-the-last-24-hours/4403/3 "2023-07-17T16:35:58Z")
Hi @Manolis_Vasilakis , thanks for the reply. I cannot use `sum` in this case, because I only expect the alarm to be raised if the chart value is **constantly** \> 0 over the last 24 hours. It is normal and expected to have a value of \> 0 for a brief period (up to 23h59), `unattended-upgrades` will run once a day, and automatically upgrade packages that need to be upgraded. Even during the automatic upgrade process itself, there is a brief period when the chart value is \> 0 (package lists have been updated but the package upgrade process is not completely finished). So using `sum` would trigger an alarm (and not clear it for 24 hours), even if no manual action is needed (the situation will resolve itself under the accepted delay). However, if the number of upgradable packages is constantly \> 0 for the last 24 hours, that may indicate a problem that needs to be looked into (a specific package fails to upgrade, or auto-upgrades from a specific third-party repository are not enabled and the package must be upgraded manually, etc). I think `lookup: min -1d of upgradable` should perfectly describe this condition in theory, correct? But as you can see, it does not work in practice. ---
**Author:** ![Manolis\_Vasilakis](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/manolis_vasilakis/32/820_2.png) [@Manolis\_Vasilakis](https://community.netdata.cloud/u/Manolis_Vasilakis)\ **Post date:** [July 17, 2023, 4:39pm UTC](https://community.netdata.cloud/t/trigger-an-alarm-when-a-chart-never-reaches-zero-over-the-last-24-hours/4403/4 "2023-07-17T16:39:01Z")
Yes, in this respect it makes sense. Indeed the `min` in this case sounds reasonable. I’ll try to reproduce a similar alert and will let you know. ---
**Author:** ![ilyam8](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/ilyam8/32/134_2.png) [@ilyam8](https://community.netdata.cloud/u/ilyam8)\ **Post date:** [July 17, 2023, 8:34pm UTC](https://community.netdata.cloud/t/trigger-an-alarm-when-a-chart-never-reaches-zero-over-the-last-24-hours/4403/5 "2023-07-17T20:34:33Z")
Hey, @nodiscc. And what value do you see in Alarms-\>All-\>your alarm? ---
**Author:** ![Manolis\_Vasilakis](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/manolis_vasilakis/32/820_2.png) [@Manolis\_Vasilakis](https://community.netdata.cloud/u/Manolis_Vasilakis)\ **Post date:** [July 18, 2023, 9:12am UTC](https://community.netdata.cloud/t/trigger-an-alarm-when-a-chart-never-reaches-zero-over-the-last-24-hours/4403/6 "2023-07-18T09:12:44Z")
Hi @nodiscc Can you try `lookup: min -1d unaligned of upgradable` ? ---
**Author:** ![nodiscc](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/nodiscc/32/2982_2.png) [@nodiscc](https://community.netdata.cloud/u/nodiscc)\ **Post date:** [July 25, 2023, 9:54am UTC](https://community.netdata.cloud/t/trigger-an-alarm-when-a-chart-never-reaches-zero-over-the-last-24-hours/4403/7 "2023-07-25T09:54:13Z")
Hi @ilyam8 I had another occurrence of this problem today, it allowed me to investigate again: - a package had an upgrade pending for several days → chart value constantly \> 0 → alarm raised → good! - changed `lookup:` to `min -1d unaligned of upgradable`, ran `sudo netdatacli reload-health` - ran `apt upgrade`, package successfully upgraded, verify that no more upgrades are available with `apt list --upgradable` → OK ![image](https://canada1.discourse-cdn.com/flex029/uploads/netdata2/original/2X/a/a9f24394d7b361862cab588bed076fad56ac5813.png) - Waited 10 minutes, expecting the alarm to clear → alarm not cleared: ![image](https://canada1.discourse-cdn.com/flex029/uploads/netdata2/original/2X/4/4d9faec3fbc0523caeefda916442d6f50bc3fbdb.png) ![image](https://canada1.discourse-cdn.com/flex029/uploads/netdata2/original/2X/a/ab00d3e0f26031ef7c6b3060de9f884045259883.png) > what value do you see in Alarms-\>All-\>your alarm? As you can see the value stays at `1` ---
**Author:** ![Manolis\_Vasilakis](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/manolis_vasilakis/32/820_2.png) [@Manolis\_Vasilakis](https://community.netdata.cloud/u/Manolis_Vasilakis)\ **Post date:** [August 1, 2023, 1:02pm UTC](https://community.netdata.cloud/t/trigger-an-alarm-when-a-chart-never-reaches-zero-over-the-last-24-hours/4403/8 "2023-08-01T13:02:07Z")
Hi @nodiscc Eventually, do you have a point where it did clear? Will check this again… ---
**Author:** ![nodiscc](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/nodiscc/32/2982_2.png) [@nodiscc](https://community.netdata.cloud/u/nodiscc)\ **Post date:** [August 1, 2023, 2:16pm UTC](https://community.netdata.cloud/t/trigger-an-alarm-when-a-chart-never-reaches-zero-over-the-last-24-hours/4403/9 "2023-08-01T14:16:25Z")
Hi, I just had the same situation: 1 package not automatically upgraded (only for a few hours), alarm raised, I manually upgraded it at 15:00~15:10 GMT+2, alarm is still active right now (16:15 GMT+2), I will check when the alarm clears, and let you know. ---
**Author:** ![nodiscc](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/nodiscc/32/2982_2.png) [@nodiscc](https://community.netdata.cloud/u/nodiscc)\ **Post date:** [August 1, 2023, 3:57pm UTC](https://community.netdata.cloud/t/trigger-an-alarm-when-a-chart-never-reaches-zero-over-the-last-24-hours/4403/10 "2023-08-01T15:57:22Z")
I checked the event log for this alarm (it is still active, ~5 hours after): ![image](https://canada1.discourse-cdn.com/flex029/uploads/netdata2/original/2X/2/2fa8e98cec6a25edf95dad4e9984825ec5cd924e.png) Here is the chart to which this alarm is attached (last 6 hours): ![image](https://canada1.discourse-cdn.com/flex029/uploads/netdata2/original/2X/5/591e6d01c043c04a4aab6f3898a027cf15039e9c.png) I will provide an update when the alarm clears. ---
**Author:** ![nodiscc](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/nodiscc/32/2982_2.png) [@nodiscc](https://community.netdata.cloud/u/nodiscc)\ **Post date:** [August 2, 2023, 9:00am UTC](https://community.netdata.cloud/t/trigger-an-alarm-when-a-chart-never-reaches-zero-over-the-last-24-hours/4403/11 "2023-08-02T09:00:36Z")
It cleared at 02:52:45, for no apparent reason (exactly 14 hours after it was first raised): ![image](https://canada1.discourse-cdn.com/flex029/uploads/netdata2/original/2X/d/d7d7e284a8e54fcc386a59047be3ff9d6affa304.png) ![image](https://canada1.discourse-cdn.com/flex029/uploads/netdata2/original/2X/3/3b1b2c1c48007a7dd4a22750e395a9f257821f19.png) ---
**Author:** ![Manolis\_Vasilakis](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/manolis_vasilakis/32/820_2.png) [@Manolis\_Vasilakis](https://community.netdata.cloud/u/Manolis_Vasilakis)\ **Post date:** [September 18, 2023, 7:57am UTC](https://community.netdata.cloud/t/trigger-an-alarm-when-a-chart-never-reaches-zero-over-the-last-24-hours/4403/12 "2023-09-18T07:57:55Z")
Hi @nodiscc From the looks of these charts, it appears the collector isn’t collecting data at all times? There are quite some gaps? That could interfere with the alert’s calculation. Could we check that first? Are you using a custom collector to gather these? ---
**Author:** ![nodiscc](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/nodiscc/32/2982_2.png) [@nodiscc](https://community.netdata.cloud/u/nodiscc)\ **Post date:** [September 18, 2023, 3:34pm UTC](https://community.netdata.cloud/t/trigger-an-alarm-when-a-chart-never-reaches-zero-over-the-last-24-hours/4403/13 "2023-09-18T15:34:30Z")
Hi, > it appears the collector isn’t collecting data at all times? There are quite some gaps? Correct, these gaps are caused by the VM/host not being on all the time. > Are you using a custom collector to gather these? Yes as mentioned in the first post this chart is generated by [GitHub - nodiscc/netdata-apt: [mirror] Check/graph number of upgradable packages - netdata plugin](https://github.com/nodiscc/netdata-apt) which I maintain. Currently you can see the alarm being active: ![image](https://canada1.discourse-cdn.com/flex029/uploads/netdata2/original/2X/a/af43aab975b6fed4a4ec7fba17122a5e0d2db0fb.png) However upgrades were applied a few hours ago (so the alarm should now be inactive): ![image](https://canada1.discourse-cdn.com/flex029/uploads/netdata2/original/2X/4/4fc47604537728aa29f5f677b28635d085f3fd49.png) Here is the current configuration for the collector: ```auto $ sudo cat /etc/netdata/python.d/apt.conf # netdata python.d.plugin configuration for apt # # This file is in YaML format. Generally the format is: # # name: value # # There are 2 sections: # - global variables # - one or more JOBS # # JOBS allow you to collect values from multiple sources. # Each source will have its own set of charts. # # JOB parameters have to be indented (using spaces only, example below). # ---------------------------------------------------------------------- # Global Variables # These variables set the defaults for all JOBs, however each JOB # may define its own, overriding the defaults. # update_every sets the default data collection frequency. # If unset, the python.d.plugin default is used. update_every: 600 # priority controls the order of charts at the netdata dashboard. # Lower numbers move the charts towards the top of the page. # If unset, the default of 90000 is used. # priority: 90000 # penalty indicates whether to apply penalty to update_every in case of failures. # Penalty will increase every 5 failed updates in a row. Maximum penalty is 10 minutes. # penalty: yes # autodetection_retry sets the job re-check interval in seconds. # The job is not deleted if check fails. # Attempts to start the job are made once every autodetection_retry. # This feature is disabled by default. # autodetection_retry: 0 # ---------------------------------------------------------------------- # JOBS (data collection sources) # # The default JOBS share the same *name*. JOBS with the same name # are mutually exclusive. Only one of them will be allowed running at # any time. This allows autodetection to try several alternatives and # pick the one that works. # # Any number of jobs is supported. # # All python.d.plugin JOBS (for all its modules) support a set of # predefined parameters. These are: # # job_name: # name: myname # the JOB's name as it will appear at the # # dashboard (by default is the job_name) # # JOBs sharing a name are mutually exclusive # update_every: 1 # the JOB's data collection frequency # priority: 60000 # the JOB's order on the dashboard # penalty: yes # the JOB's penalty # autodetection_retry: 0 # the JOB's re-check interval in seconds # # This module does not provide any additional option. ``` And the health configuration: ```auto $ sudo cat /etc/netdata/health.d/apt.conf alarm: apt_upgradable on: apt.upgradable lookup: min -1d of upgradable every: 60s warn: $this > 0 units: packages info: packages with available upgrades to: sysadmin class: Errors alarm: apt_distribution_version on: apt.distribution_version calc: $distribution_version every: 60s warn: $this < 12 crit: $this < 11 units: distribution version info: distribution upgrade available to: sysadmin class: Errors alarm: distribution_version_error on: apt.distribution_version_error calc: $distribution_version_error every: 60s warn: $this > 0 units: apt failed check info: state file was unreadable to: sysadmin class: Errors ``` ---
**Author:** ![Manolis\_Vasilakis](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/manolis_vasilakis/32/820_2.png) [@Manolis\_Vasilakis](https://community.netdata.cloud/u/Manolis_Vasilakis)\ **Post date:** [September 19, 2023, 12:40pm UTC](https://community.netdata.cloud/t/trigger-an-alarm-when-a-chart-never-reaches-zero-over-the-last-24-hours/4403/14 "2023-09-19T12:40:15Z")
Hi @nodiscc Thanks. I’ll setup them on a Debian test node and will try to debug. ---
**Author:** ![Manolis\_Vasilakis](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/manolis_vasilakis/32/820_2.png) [@Manolis\_Vasilakis](https://community.netdata.cloud/u/Manolis_Vasilakis)\ **Post date:** [September 21, 2023, 1:07pm UTC](https://community.netdata.cloud/t/trigger-an-alarm-when-a-chart-never-reaches-zero-over-the-last-24-hours/4403/15 "2023-09-21T13:07:09Z")
So, I’ve been testing this for a couple of days, and indeed there is an issue. `unaligned` in lookup seems to give better results. One guess perhaps is that the problem lies with the 600 update every of the collector. Could you perhaps test with e.g. `60` ? ---
**Author:** ![Manolis\_Vasilakis](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/manolis_vasilakis/32/820_2.png) [@Manolis\_Vasilakis](https://community.netdata.cloud/u/Manolis_Vasilakis)\ **Post date:** [September 21, 2023, 3:20pm UTC](https://community.netdata.cloud/t/trigger-an-alarm-when-a-chart-never-reaches-zero-over-the-last-24-hours/4403/16 "2023-09-21T15:20:17Z")
Another possibility that we think might be the cause, is that it gets the values from another tier than the first one (because of the `-1d`), leading to not being actually able to detect a 0 value (since the values from other tiers are averaged). A possible solution would be to force health to run alerts from values in tier 0 only. I’ll try to check and update. Edit: Could you check if the alert works better with latest nightlies? Thanks! ---
**Author:** ![nodiscc](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/nodiscc/32/2982_2.png) [@nodiscc](https://community.netdata.cloud/u/nodiscc)\ **Post date:** [October 11, 2023, 1:37pm UTC](https://community.netdata.cloud/t/trigger-an-alarm-when-a-chart-never-reaches-zero-over-the-last-24-hours/4403/17 "2023-10-11T13:37:40Z")
> `unaligned` in lookup seems to give better results. I’ve updated the lookup expression on all my hosts and will keep an eye on the results. > the 600 update every of the collector. Could you perhaps test with e.g. `60` ? I’d rather avoid rather these costly checks to run too often (a maximum 10 minute lag is acceptable), but I have switched to 60s on a few hosts and will keep an eye on the results. > Another possibility that we think might be the cause, is that it gets the values from another tier than the first one (because of the `-1d`), leading to not being actually able to detect a 0 value (since the values from other tiers are averaged). > Could you check if the alert works better with latest nightlies? I deploy from Debian packages from [netdata/netdata - Packages · packagecloud](https://packagecloud.io/netdata/netdata) so I will have to wait for the next release (which will give me some time to evaluate the changes I already made). I will keep you updated. Thanks again for your help ---
**Author:** ![nodiscc](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/nodiscc/32/2982_2.png) [@nodiscc](https://community.netdata.cloud/u/nodiscc)\ **Post date:** [October 25, 2023, 8:40pm UTC](https://community.netdata.cloud/t/trigger-an-alarm-when-a-chart-never-reaches-zero-over-the-last-24-hours/4403/18 "2023-10-25T20:40:01Z")
Hi, The problem now seems to be resolved by adding `unaligned` to the lookup expression (`lookup: min -1d unaligned of upgradable`) on netdata 1.43.0. The collector `update every` frequency has been kept to 600, the alarm `every` has been kept to 60s. The alarm is cleared immediately when the collector updates the chart and the new minimum value becomes `0`. Thanks @Manolis_Vasilakis @ilyam8 for your help. As a last request, where can I read more about the `unaligned` operator? I could not find anything on [Configure alerts | Learn Netdata](https://learn.netdata.cloud/docs/alerting/health-configuration-reference) Thanks again! ---
**Author:** ![Manolis\_Vasilakis](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/manolis_vasilakis/32/820_2.png) [@Manolis\_Vasilakis](https://community.netdata.cloud/u/Manolis_Vasilakis)\ **Post date:** [October 31, 2023, 1:12pm UTC](https://community.netdata.cloud/t/trigger-an-alarm-when-a-chart-never-reaches-zero-over-the-last-24-hours/4403/19 "2023-10-31T13:12:30Z")
Hi @nodiscc glad to see it worked. For `unaligned`, there is some more information here → [Database queries/lookup | Learn Netdata](https://learn.netdata.cloud/docs/rest-api/queries#time-frame-alignment)