Getting Prometheus errors for Telegram bot

Help
1

System info

  • uname -r: 5.4.0-58-generic
  • netdata -v: v1.26.0-404-nightly
  • distro: Ubuntu 20.04.1 LTS

Netdata config

The problem

I tried to set up a Telegram bot as notification service by following the page in the docs.

So I changed the contents of /usr/libexec/netdata/plugins.d/alarm-notify.sh to this:

# telegram configs
TELEGRAM_BOT_TOKEN="TOKEN"
DEFAULT_RECIPIENT_TELEGRAM="ID"
SEND_TELEGRAM="YES"

and also changed /usr/lib/netdata/conf.d/health_alarm_notify.conf:

# telegram configs
TELEGRAM_BOT_TOKEN="TOKEN"
DEFAULT_RECIPIENT_TELEGRAM="ID"
SEND_TELEGRAM="YES"

then I restarted the netdata unit with sudo systemctl restart netdata and despite receiving email notifications, 0 notifications got sent to telegram.

Logs

I checked the error logs:

cat /var/log/netdata/error.log | grep -i telegram

and noticed this:

error.log 

2020-12-17 19:56:02: go.d ERROR: prometheus[telegram_bot_for_alertmanager_local] Get "http://127.0.0.1:9087/metrics": dial tcp 127.0.0.1:9087: connect: connection refused
2020-12-17 19:56:02: go.d ERROR: prometheus[telegram_bot_for_alertmanager_local] check failed
2020-12-17 20:02:35: go.d ERROR: prometheus[telegram_bot_for_alertmanager_local] Get "http://127.0.0.1:9087/metrics": dial tcp 127.0.0.1:9087: connect: connection refused
2020-12-17 20:02:35: go.d ERROR: prometheus[telegram_bot_for_alertmanager_local] check failed
2020-12-17 20:13:47: go.d ERROR: prometheus[telegram_bot_for_alertmanager_local] Get "http://127.0.0.1:9087/metrics": dial tcp 127.0.0.1:9087: connect: connection refused
2020-12-17 20:13:47: go.d ERROR: prometheus[telegram_bot_for_alertmanager_local] check failed
2020-12-17 20:47:21: go.d ERROR: prometheus[telegram_bot_for_alertmanager_local] Get "http://127.0.0.1:9087/metrics": dial tcp 127.0.0.1:9087: connect: connection refused
2020-12-17 20:47:21: go.d ERROR: prometheus[telegram_bot_for_alertmanager_local] check failed
2020-12-18 08:18:24: go.d ERROR: prometheus[telegram_bot_for_alertmanager_local] Get "http://127.0.0.1:9087/metrics": dial tcp 127.0.0.1:9087: connect: connection refused
2020-12-18 08:18:24: go.d ERROR: prometheus[telegram_bot_for_alertmanager_local] check failed
2020-12-18 09:53:06: go.d ERROR: prometheus[telegram_bot_for_alertmanager_local] Get "http://127.0.0.1:9087/metrics": dial tcp 127.0.0.1:9087: connect: connection refused
2020-12-18 09:53:06: go.d ERROR: prometheus[telegram_bot_for_alertmanager_local] check failed
2020-12-18 09:59:41: go.d ERROR: prometheus[telegram_bot_for_alertmanager_local] Get "http://127.0.0.1:9087/metrics": dial tcp 127.0.0.1:9087: connect: connection refused
2020-12-18 09:59:41: go.d ERROR: prometheus[telegram_bot_for_alertmanager_local] check failed
2020-12-18 10:02:05: go.d ERROR: prometheus[telegram_bot_for_alertmanager_local] Get "http://127.0.0.1:9087/metrics": dial tcp 127.0.0.1:9087: connect: connection refused
2020-12-18 10:02:05: go.d ERROR: prometheus[telegram_bot_for_alertmanager_local] check failed
2020-12-18 10:11:08: go.d ERROR: prometheus[telegram_bot_for_alertmanager_local] Get "http://127.0.0.1:9087/metrics": dial tcp 127.0.0.1:9087: connect: connection refused
2020-12-18 10:11:08: go.d ERROR: prometheus[telegram_bot_for_alertmanager_local] check failed
2020-12-18 10:12:52: go.d ERROR: prometheus[telegram_bot_for_alertmanager_local] Get "http://127.0.0.1:9087/metrics": dial tcp 127.0.0.1:9087: connect: connection refused
2020-12-18 10:12:52: go.d ERROR: prometheus[telegram_bot_for_alertmanager_local] check failed
2020-12-18 10:21:36: go.d ERROR: prometheus[telegram_bot_for_alertmanager_local] Get "http://127.0.0.1:9087/metrics": dial tcp 127.0.0.1:9087: connect: connection refused
2020-12-18 10:21:36: go.d ERROR: prometheus[telegram_bot_for_alertmanager_local] check failed
2020-12-18 10:22:29: go.d ERROR: prometheus[telegram_bot_for_alertmanager_local] Get "http://127.0.0.1:9087/metrics": dial tcp 127.0.0.1:9087: connect: connection refused
2020-12-18 10:22:29: go.d ERROR: prometheus[telegram_bot_for_alertmanager_local] check failed
2020-12-18 10:26:28: go.d ERROR: prometheus[telegram_bot_for_alertmanager_local] Get "http://127.0.0.1:9087/metrics": dial tcp 127.0.0.1:9087: connect: connection refused
2020-12-18 10:26:28: go.d ERROR: prometheus[telegram_bot_for_alertmanager_local] check failed

it seems like something doesn’t work in prometheus but I am not sure how to fix it.

2

Looks like your prometheus metrics URL is down or the Netdata Agent is unable to connect to it.

3

oh my bad, I forgot to install prometheus

after installing it, it instantly worked

thanks :smiley:

4

Hey @talentlessguy,

I removed the termbin url because it now leads to 404. If you want, you can repaste the configuration between two groups of three backticks (`).