using wildcards in alarms

Help
, ,
1

I have the following alarm that’s working:

   alarm: vault_sealed
      on: statsd_vault.core.unsealed.vault-cluster-abcde123_gauge
  lookup: average -5s
   units: %
   every: 1s
    warn: $this > 0
    crit: $this > 0
 summary: vault is sealed! MANUAL INTERVENTION REQUIRED!
    info: contact vault admins to perform the unsealing ceremony
      to: sysadmin

(N.B. the vault is unsealed, so this will always trigger, and am just using this to troubleshoot the context)

however, I’d like to use a wildcard in the context statement to make it more generic. something like,

   alarm: vault_sealed
      on: statsd_vault.core.unsealed.*
  lookup: average -5s
   units: %
   every: 1s
    warn: $this > 0
    crit: $this > 0
 summary: vault is sealed! MANUAL INTERVENTION REQUIRED!
    info: contact vault admins to perform the unsealing ceremony
      to: sysadmin

I’ve searched the documentation and found the simple patterns, but either I’m not applying it correctly or it’s not supported for alarms/templates?

Any thoughts or ideas?

TIA :slight_smile:

2

Hey, @enkaskal.

   alarm: vault_sealed
      on: statsd_vault.core.unsealed.*
  • on is applied to the id of the chart when using alarm
  • on is applied to the context of charts when using template.

Do “statsd_vault.core.unsealed.*” charts have the same context?

This is where you can find it:

Screenshot 2024-08-23 at 11.05.50
Screenshot 2024-08-23 at 11.05.502492×886 184 KB

3

Hi ilyam8, thanks for the reply :slight_smile:

Here is the chart of the actual instance I’m testing:

Screenshot_20240823_020523
Screenshot_20240823_020523776×293 16.7 KB

This works and generates an alarm in the dashboard:

   alarm: vault_sealed
      on: statsd_vault.core.unsealed.vault-cluster-bb4c27a6_gauge
  lookup: average -5s
   units: %
   every: 1s
    warn: $this > 0
    crit: $this > 0
 summary: vault is sealed! MANUAL INTERVENTION REQUIRED!
    info: contact vault admins to perform the unsealing ceremony
      to: sysadmin

as well as a log of:

Aug 23 02:37:15 netdata netdata[143588]: ALERT 'vault_sealed' of instance 'statsd_vault.core.unsealed.vault-cluster-bb4c27a6_gauge' on node 'vault', transitioned from UNINITIALIZED to CRITICAL

as does this:

template: vault_sealed
      on: statsd_vault.core.unsealed.vault-cluster-bb4c27a6_gauge
  lookup: average -5s
   units: %
   every: 1s
    warn: $this > 0
    crit: $this > 0
 summary: vault is sealed! MANUAL INTERVENTION REQUIRED!
    info: contact vault admins to perform the unsealing ceremony
      to: sysadmin

However, neither of these work:

template: vault_sealed
      on: statsd_vault.core.unsealed.*
  lookup: average -5s
   units: %
   every: 1s
    warn: $this > 0
    crit: $this > 0
 summary: vault is sealed! MANUAL INTERVENTION REQUIRED!
    info: contact vault admins to perform the unsealing ceremony
      to: sysadmin

   alarm: vault_sealed
      on: statsd_vault.core.unsealed.*
  lookup: average -5s
   units: %
   every: 1s
    warn: $this > 0
    crit: $this > 0
 summary: vault is sealed! MANUAL INTERVENTION REQUIRED!
    info: contact vault admins to perform the unsealing ceremony
      to: sysadmin

and just for completeness I tried these too:

   alarm: vault_sealed
      on: vault.core.unsealed.*
  lookup: average -5s
   units: %
   every: 1s
    warn: $this > 0
    crit: $this > 0
 summary: vault is sealed! MANUAL INTERVENTION REQUIRED!
    info: contact vault admins to perform the unsealing ceremony
      to: sysadmin

template: vault_sealed
      on: vault.core.unsealed.*
  lookup: average -5s
   units: %
   every: 1s
    warn: $this > 0
    crit: $this > 0
 summary: vault is sealed! MANUAL INTERVENTION REQUIRED!
    info: contact vault admins to perform the unsealing ceremony
      to: sysadmin

but, of course, they didn’t work either.

4

@enkaskal on doesn’t support wildcards, it must be an exact value. We need to extend Netdata’s Statsd protocol to support tags (labels).

5

@ilyam8 gotcha; thank you!

did you want me to file a feature request somewhere or vote for it somewhere?

6

did you want me to file a feature request

Yes, please.

7

@ilyam8 np :slight_smile: where at? somewhere here in the forums or on github? (or both?)

EDIT: grammar

8

GitHub Feature Request

9

filed and @'d you on github: [Feat]: extend Netdata’s Statsd protocol to support tags (labels). · Issue #18412 · netdata/netdata · GitHub

thanks for the assistance; i really appreciate it! :smiley: