Telegram not receiving some alerts while gotify works ok

[enrirollins]

Problem/Question

Hi,

I start by saying I’ve read the following topic

Not all alarms send to the Telegram

Hello everybody! I cannot understand why not all alarms are sent to my telegram. When i used centralized alarm notifications via email i got all notifications from alarms log. After that i disabled centralized alarm notifications of Netdata Cloud and setup Netdata Agent telegram notifications by that documentation section - Telegram | Learn Netdata. I left all the health configuration by default. And it’s working fine, but i only get system.ram alarms, but in the dashboard alarms logs I still s…

But I couldn’t relate the answers to what I’m looking for.
I activated two alerts’ channels

• telegram bot
• gotify

Some alerts are not sent to the telegram bot I’ve created while gotify is working every time
These are my config:

SEND_TELEGRAM="YES"
TELEGRAM_BOT_TOKEN="SECRET"
TELEGRAM_RETRIES_ON_LIMIT="5"
DEFAULT_RECIPIENT_TELEGRAM="SECRET"

SEND_GOTIFY="YES"
GOTIFY_APP_TOKEN="SECRET"
GOTIFY_APP_URL="https://SECRET"
DEFAULT_RECIPIENT_GOTIFY="root"

The test alerts and some other are working:

But others are not, only gotify’s channel is, for example:

Environment/Browser/Agent’s version etc

One Debian 11 parent node is taking care of the alerts, the netdata version is the following:
netdata v1.44.0-144-g9638fddbd
static

I can obviously also post the whole buildinfo if needed.

What could the problem be?

Thanks!

Enrico

[Thiago_Marques_0]

Hello @enrirollins ,

I want to confirm something with you: are you using a negative number with your DEFAULT_RECIPIENT_TELEGRAM? What is the HTTP response you receive when you run alarm-notify.sh test?

Best regards,

Thiago

[enrirollins]

Hi @Thiago_Marques_0

I’m not using a negative number as DEFAULT_RECIPIENT_TELEGRAM and here’s the HTTP response to the testing WARNING alert:

# SENDING TEST WARNING ALARM TO ROLE: sysadmin
time=2024-01-12T08:42:24.933+01:00 comm=alarm-notify.sh source=health level=debug tid=56617 thread=alarm-notify msg_id=6db0018e83e34320ae2a659d78019fb7 node=SECRET instance=test.chart alert_id=1 alert_unique_id=1 alert=test_alarm alert_class=Test alert_recipient=sysadmin alert_duration=1 alert_value=100 alert_value_old=90 alert_status=WARNING alert_value_old=CLEAR alert_units=units alert_summary="a test alarm" alert_info="this is a test alarm to verify notifications work" request="'/opt/netdata/usr/libexec/netdata/plugins.d/alarm-notify.sh' 'sysadmin' 'SECRET' '1' '1' '1' '1705045344' 'test_alarm' 'test.chart' 'WARNING' 'CLEAR' '100' '90' '/opt/netdata/usr/libexec/netdata/plugins.d/alarm-notify.sh' '1' '1' 'units' 'this is a test alarm to verify notifications work' 'new value' 'old value' 'evaluated expression' 'expression variable values' '0' '0' '' '' 'Test' 'command to edit the alarm=0=SECRET' '' '' 'a test alarm' " msg="[ALERT NOTIFICATION]: Loading config file '/opt/netdata/usr/lib/netdata/conf.d/health_alarm_notify.conf'..."
time=2024-01-12T08:42:24.970+01:00 comm=alarm-notify.sh source=health level=debug tid=56618 thread=alarm-notify msg_id=6db0018e83e34320ae2a659d78019fb7 node=SECRET instance=test.chart alert_id=1 alert_unique_id=1 alert=test_alarm alert_class=Test alert_recipient=sysadmin alert_duration=1 alert_value=100 alert_value_old=90 alert_status=WARNING alert_value_old=CLEAR alert_units=units alert_summary="a test alarm" alert_info="this is a test alarm to verify notifications work" request="'/opt/netdata/usr/libexec/netdata/plugins.d/alarm-notify.sh' 'sysadmin' 'SECRET' '1' '1' '1' '1705045344' 'test_alarm' 'test.chart' 'WARNING' 'CLEAR' '100' '90' '/opt/netdata/usr/libexec/netdata/plugins.d/alarm-notify.sh' '1' '1' 'units' 'this is a test alarm to verify notifications work' 'new value' 'old value' 'evaluated expression' 'expression variable values' '0' '0' '' '' 'Test' 'command to edit the alarm=0=SECRET' '' '' 'a test alarm' " msg="[ALERT NOTIFICATION]: Loading config file '/opt/netdata/etc/netdata/health_alarm_notify.conf'..."
--- BEGIN curl command ---
/usr/bin/curl --data-urlencode parse_mode=HTML --data-urlencode disable_web_page_preview=true --data-urlencode $'text=\342\232\240\357\270\217 SECRET needs attention - <b>test alarm</b>\ntest.chart\n<a href="https://registry.my-netdata.io/registry-alert-redirect.html?agent_machine_guid=ffaa11cc-27d3-11ee-8017-a8a159053b7e&host_machine_guid=&transition_id=&host=SECRET&chart=test.chart&alarm=test_alarm&alarm_unique_id=1&alarm_id=1&alarm_event_id=1&alarm_when=1705045344&alarm_status=WARNING&alarm_chart=test.chart&alarm_value=new%20value">a test alarm = new value</a>\n<i>this is a test alarm to verify notifications work</i>' https://api.telegram.org/bot6921605782:AAGp0KqA26-SaF_E7HAACBax628VbqZtXI4/sendMessage\?chat_id=224664831
--- END curl command ---
--- BEGIN received response ---
{"ok":true,"result":{"message_id":324,"from":{"id":6921605782,"is_bot":true,"first_name":"SECRET","username":"SECRET"},"chat":{"id":SECRET,"first_name":"enrico","type":"private"},"date":1705045345,"text":"\u26a0\ufe0f SECRET needs attention - test alarm\ntest.chart\na test alarm = new value\nthis is a test alarm to verify notifications work","entities":[{"offset":41,"length":10,"type":"bold"},{"offset":63,"length":24,"type":"text_link","url":"https://registry.my-netdata.io/registry-alert-redirect.html?agent_machine_guid=ffaa11cc-27d3-11ee-8017-a8a159053b7e&host_machine_guid=&transition_id=&host=SECRET&chart=test.chart&alarm=test_alarm&alarm_unique_id=1&alarm_id=1&alarm_event_id=1&alarm_when=1705045344&alarm_status=WARNING&alarm_chart=test.chart&alarm_value=new%20value"},{"offset":88,"length":49,"type":"italic"}],"link_preview_options":{"is_disabled":true}}}
--- END received response ---
RECEIVED HTTP RESPONSE CODE: 200
time=2024-01-12T08:42:25.881+01:00 comm=alarm-notify.sh source=health level=info tid=56643 thread=alarm-notify msg_id=6db0018e83e34320ae2a659d78019fb7 node=SECRET instance=test.chart alert_id=1 alert_unique_id=1 alert=test_alarm alert_class=Test alert_recipient=sysadmin alert_duration=1 alert_value=100 alert_value_old=90 alert_status=WARNING alert_value_old=CLEAR alert_units=units alert_summary="a test alarm" alert_info="this is a test alarm to verify notifications work" request="'/opt/netdata/usr/libexec/netdata/plugins.d/alarm-notify.sh' 'sysadmin' 'SECRET' '1' '1' '1' '1705045344' 'test_alarm' 'test.chart' 'WARNING' 'CLEAR' '100' '90' '/opt/netdata/usr/libexec/netdata/plugins.d/alarm-notify.sh' '1' '1' 'units' 'this is a test alarm to verify notifications work' 'new value' 'old value' 'evaluated expression' 'expression variable values' '0' '0' '' '' 'Test' 'command to edit the alarm=0=SECRET' '' '' 'a test alarm' " msg="[ALERT NOTIFICATION]: sent telegram notification to '224664831' for notification to 'sysadmin' for transition from CLEAR to WARNING, of alert 'test_alarm' = 'new value', of instance 'test.chart', context '' on host 'SECRET'"
--- BEGIN curl command ---
/usr/bin/curl -X POST -H Content-Type:\ application/json -d $'  {\n    "title" : "WARNING, test_alarm = new value, on SECRET",\n    "message" : "Fri Jan 12 08:42:24 CET 2024: test.chart new value",\n    "priority" : 4\n  }' SECRET_GOTIFY_URL_AND_TOKEN
--- END curl command ---
--- BEGIN received response ---
{"id":435,"appid":2,"message":"Fri Jan 12 08:42:24 CET 2024: test.chart new value","title":"WARNING, test_alarm = new value, on SECRET","priority":4,"date":"2024-01-12T08:42:26.128057155+01:00"}
--- END received response ---
RECEIVED HTTP RESPONSE CODE: 200
time=2024-01-12T08:42:26.145+01:00 comm=alarm-notify.sh source=health level=info tid=56654 thread=alarm-notify msg_id=6db0018e83e34320ae2a659d78019fb7 node=SECRET instance=test.chart alert_id=1 alert_unique_id=1 alert=test_alarm alert_class=Test alert_recipient=sysadmin alert_duration=1 alert_value=100 alert_value_old=90 alert_status=WARNING alert_value_old=CLEAR alert_units=units alert_summary="a test alarm" alert_info="this is a test alarm to verify notifications work" request="'/opt/netdata/usr/libexec/netdata/plugins.d/alarm-notify.sh' 'sysadmin' 'SECRET' '1' '1' '1' '1705045344' 'test_alarm' 'test.chart' 'WARNING' 'CLEAR' '100' '90' '/opt/netdata/usr/libexec/netdata/plugins.d/alarm-notify.sh' '1' '1' 'units' 'this is a test alarm to verify notifications work' 'new value' 'old value' 'evaluated expression' 'expression variable values' '0' '0' '' '' 'Test' 'command to edit the alarm=0=SECRET' '' '' 'a test alarm' " msg="[ALERT NOTIFICATION]: sent gotify event for notification to 'sysadmin' for transition from CLEAR to WARNING, of alert 'test_alarm' = 'new value', of instance 'test.chart', context '' on host 'SECRET'"
# OK

I’ve noticed that the cloud email alert works the same way so when a OUT_OF_DISK_SPACE_TIME (example) alert fires:

• gotify notification OK
• telegram KO
• cloud email KO

Thanks!

[Thiago_Marques_0]

Hello,

While I was preparing an environment to use telegram, I observed that the steps to configure Telegram 4.14.4 is different of previous versions that I configured. I am going to update it as soon I finish this answer for you.

To configure my telgram locally, I followed the steps:

1 - I create a bot accessing the @BotFather
2 - I create a new group and I invited my bot an @myidbot to it.
3 - After this, I ran the command /getgroupid@myidbot inside the group I created to get the id.
4 - Now it is time to modify netdata, I modified my /etc/netdata/health_alarm_notify.conf adding
TELEGRAM_BOT_TOKEN=“AAAAAAAA:BBBB-A_SECRET_VALUE”
DEFAULT_RECIPIENT_TELEGRAM=“-SECRET”

The TELEGRAM_BOT_TOKEN was delivered by BotFather. While DEFAULT_RECIPIENT_TELEGRAM was delivered by /getgroupid@myidbot`

5 - Finally, I ran the command /usr/libexec/netdata/plugins.d/alarm-notify.sh test and I got:

time=2024-01-12T15:50:59.393+00:00 comm=alarm-notify.sh source=health level=info tid=9035 thread=alarm-notify msg_id=6db0018e83e34320ae2a659d78019fb7 node=hades instance=test.chart alert_id=1 alert_unique_id=1 alert=test_alarm alert_class=Test alert_recipient=sysadmin alert_duration=1 alert_value=100 alert_value_old=90 alert_status=WARNING alert_value_old=CLEAR alert_units=units alert_summary="a test alarm" alert_info="this is a test alarm to verify notifications work" request="'./alarm-notify.sh' 'sysadmin' 'hades' '1' '1' '1' '1705074657' 'test_alarm' 'test.chart' 'WARNING' 'CLEAR' '100' '90' './alarm-notify.sh' '1' '1' 'units' 'this is a test alarm to verify notifications work' 'new value' 'old value' 'evaluated expression' 'expression variable values' '0' '0' '' '' 'Test' 'command to edit the alarm=0=hades' '' '' 'a test alarm' " msg="[ALERT NOTIFICATION]: sent telegram notification to '-SECRET' for notification to 'sysadmin' for transition from CLEAR to WARNING, of alert 'test_alarm' = 'new value', of instance 'test.chart', context '' on host 'hades'"

As you can see, my telegram notifications are completely different of yours. The message format that I have for Slack and Telegram arre more or less the same.

I am going to run real tests now while I am updating our documentation.
Please, let me know if your steps to configure were like mine, because this can be the source of issues.

Best regards!

[enrirollins]

hi,

the steps I took while configurating the bot are different becasue I haven’t create a group and then invited my bot to that group, I’ve instead opened a chat with the bot directly and then issued a /getid command @idbot.
That’s the ID I’m using as DEFAULT_RECIPIENT_TELEGRAM.
What sounds strange to me is that even the cloud email notifications acts exatcly like the telegram one while the gotify ones acts like the parent node email notifications.

Thanks again

Enrico