Notification suddenly stops working about checkmk-telegram-plus HOT 27 CLOSED

Do you experience this problem frequently? Have you already tried restarting the service? I have only had this issue once in 5 months now. This problem is related to the fact that the file system event handler suddenly stops responding for some reason. The only solution I can see is to move the queue to another system, because I don't really understand where something is going wrong on the Linux level.

from checkmk-telegram-plus.

As this problem has already happened to me once, an update was already on the way. I have now accelerated this through your issue, and it is now online. Can you install this update and tell me whether it works for you this way or not? Thank you.

from checkmk-telegram-plus.

Thanks for the update.

I updated telegram plus and since then the service is not able to start.

root@CheckMK:/etc/systemd/system# systemctl status checkmk-telegram-plus-monitoring.service

• checkmk-telegram-plus-monitoring.service - CheckMK Telegram Plus Service
  Loaded: loaded (/etc/systemd/system/checkmk-telegram-plus-monitoring.service; enabled; vendor preset: enabled)
  Active: failed (Result: signal) since Thu 2023-07-27 10:40:38 CEST; 4min 25s ago
  Process: 9480 ExecStartPre=/bin/sleep 60 (code=killed, signal=TERM)
  CPU: 876us

Jul 27 10:40:18 CheckMK systemd[1]: Starting CheckMK Telegram Plus Service...
Jul 27 10:40:38 CheckMK systemd[1]: checkmk-telegram-plus-monitoring.service: Control process exited, code=killed, status=15/TERM
Jul 27 10:40:38 CheckMK systemd[1]: checkmk-telegram-plus-monitoring.service: Failed with result 'signal'.
Jul 27 10:40:38 CheckMK systemd[1]: Stopped CheckMK Telegram Plus Service.
root@CheckMK:/etc/systemd/system# systemctl start checkmk-telegram-plus-monitoring.service

I also completely removed the service and reinstalled it - no change...

Checkmk is now running on version 2.2.0.p7

from checkmk-telegram-plus.

Did you also delete the old installation file and download it again? Because this has also changed. Because it works for me. Otherwise, I can still check it.

It could also be that your python version has a different numpy version. What happens when you run: python3 ~/local/share/checkmk-telegram-plus with the OMD user?

If you run the following command with the OMD user: python3 --version
What do you get as output

from checkmk-telegram-plus.

I removed everything inclouding the notification rule in checkmk, install.sh and all the folders you mentioned.
Then I reinstalled it - again not working...
Then I restored the latest backup -> all working fine again...

root@CheckMK:/tmp# systemctl status checkmk-telegram-plus-monitoring.service

• checkmk-telegram-plus-monitoring.service - CheckMK Telegram Plus Service
  Loaded: loaded (/etc/systemd/system/checkmk-telegram-plus-monitoring.service; enabled; vendor preset: enabled)
  Active: active (running) since Thu 2023-07-27 11:45:01 CEST; 4min 15s ago
  Process: 111 ExecStartPre=/bin/sleep 60 (code=exited, status=0/SUCCESS)
  Main PID: 1125 (runuser)
  Tasks: 0 (limit: 76564)
  Memory: 1.0M
  CPU: 4ms
  CGroup: /system.slice/checkmk-telegram-plus-monitoring.service
  > 1125 /usr/sbin/runuser -l monitoring -c cd /omd/sites/monitoring/local/share/checkmk-telegram-plus && /omd/sites/monitoring/bin/python3 /omd/sites/monitoring/local/share/checkmk-telegram-plus/telegram_bot.py

Installed the update and service is broken again:

root@CheckMK:/home# systemctl status checkmk-telegram-plus-monitoring.service

• checkmk-telegram-plus-monitoring.service - CheckMK Telegram Plus Service
  Loaded: loaded (/etc/systemd/system/checkmk-telegram-plus-monitoring.service; enabled; vendor preset: enabled)
  Active: activating (start-pre) since Thu 2023-07-27 12:04:19 CEST; 20s ago
  Cntrl PID: 8134 (sleep)
  Tasks: 1 (limit: 76564)
  Memory: 132.0K
  CPU: 698us
  CGroup: /system.slice/checkmk-telegram-plus-monitoring.service
  `-8134 /bin/sleep 60

Jul 27 12:04:19 CheckMK systemd[1]: Starting CheckMK Telegram Plus Service...

The output of your two commands:

root@CheckMK:/tmp# python3 ~/local/share/checkmk-telegram-plus-monitoring
python3: can't open file '/root/local/share/checkmk-telegram-plus-monitoring': [Errno 2] No such file or directory

I also tried: (with .local)
root@CheckMK:/tmp# python3 ~/.local/share/checkmk-telegram-plus-monitoring
python3: can't open file '/root/.local/share/checkmk-telegram-plus-monitoring': [Errno 2] No such file or directory

root@CheckMK:/tmp# python3 --version
Python 3.9.2

from checkmk-telegram-plus.

So 2 mistakes you made when executing the commands.

1. you have to execute it as said with the OMD user. I think you have the name of the OMD page monitoring? So try the following:

su - monitoring
cd cd local/share/checkmk-telegram-plus/
python3 telegram_bot.py
python3 --version

And 2. in the command I wrote (python3 ~/local/share/checkmk-telegram-plus) Please don't add -monitoring at the end. Just follow 1:1 the instructions above and it should work.

from checkmk-telegram-plus.

here is the output of the commands:

OMD[monitoring]:$ cd local/share/checkmk-telegram-plus/
OMD[monitoring]:/local/share/checkmk-telegram-plus$ python3 telegram_bot.py
Traceback (most recent call last):
File "/opt/omd/sites/monitoring/local/share/checkmk-telegram-plus/telegram_bot.py", line 12, in
import fqueue
File "/opt/omd/sites/monitoring/local/share/checkmk-telegram-plus/fqueue.py", line 5, in
import pandas as pd
File "/opt/omd/sites/monitoring/local/share/checkmk-telegram-plus/pandas/init.py", line 16, in
raise ImportError(
ImportError: Unable to import required dependencies:
numpy:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

• The Python version is: Python3.11 from "/omd/sites/monitoring/bin/python3"
• The NumPy version is: "1.25.1"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: No module named 'numpy.core._multiarray_umath'

OMD[monitoring]:~/local/share/checkmk-telegram-plus$ python3 --version
Python 3.11.2

from checkmk-telegram-plus.

Okay, just as I thought. There is a problem with the Numpy version... I think an update might cause this for several people through various environments, I'll see if I can find a way to prevent it. I'll get back to you as soon as I have an update ready.

from checkmk-telegram-plus.

Okay, I have updated it now. Can you please uninstall and install it again? Thank you. I also tested it in my lab and it worked there without any problems.

from checkmk-telegram-plus.

Thanks a lot man - issue is fixed - menu is back and quering the status works again - I hope the notification is now back to normal. I will keep you in the picture...

Best regards
Dr Icetea

from checkmk-telegram-plus.

For the automatic notification, you can try triggering a custom notification. To do this, open a host that triggers your notification rule. Then click on "Commands" in the view and then on "Custom Notification". A small window should appear. There, you click on "Send". If everything works, you should receive a message from the bot fairly soon (after +/- 20 seconds).

If this works, I would suggest we wait a few days. If you no longer have the problem, I would kindly request you (if you no longer have the problem) to inform me of this here, because there is nothing more we can do now than wait and see.

from checkmk-telegram-plus.

too early - the notification does not work - the custom notification has triggered many mails but none via Telegram...

here is the output of the python3 telegram_bot.py - hope this helps

OMD[monitoring]:~/local/share/checkmk-telegram-plus$ python3 telegram_bot.py
Error while getting Updates: Conflict: terminated by other getUpdates request; make sure that only one bot instance is running
No error handlers are registered, logging exception.
Traceback (most recent call last):
File "/opt/omd/sites/monitoring/local/share/checkmk-telegram-plus/telegram/ext/_updater.py", line 607, in _network_loop_retry
if not await action_cb():
^^^^^^^^^^^^^^^^^
File "/opt/omd/sites/monitoring/local/share/checkmk-telegram-plus/telegram/ext/_updater.py", line 335, in polling_action_cb
raise exc
File "/opt/omd/sites/monitoring/local/share/checkmk-telegram-plus/telegram/ext/_updater.py", line 320, in polling_action_cb
updates = await self.bot.get_updates(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/omd/sites/monitoring/local/share/checkmk-telegram-plus/telegram/ext/_extbot.py", line 524, in get_updates
updates = await super().get_updates(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/omd/sites/monitoring/local/share/checkmk-telegram-plus/telegram/_bot.py", line 331, in decorator
result = await func(*args, **kwargs) # skipcq: PYL-E1102
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/omd/sites/monitoring/local/share/checkmk-telegram-plus/telegram/_bot.py", line 3510, in get_updates
await self._post(
File "/opt/omd/sites/monitoring/local/share/checkmk-telegram-plus/telegram/_bot.py", line 419, in _post
return await self._do_post(
^^^^^^^^^^^^^^^^^^^^
File "/opt/omd/sites/monitoring/local/share/checkmk-telegram-plus/telegram/ext/_extbot.py", line 306, in _do_post
return await super()._do_post(
^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/omd/sites/monitoring/local/share/checkmk-telegram-plus/telegram/_bot.py", line 450, in _do_post
return await request.post(
^^^^^^^^^^^^^^^^^^^
File "/opt/omd/sites/monitoring/local/share/checkmk-telegram-plus/telegram/request/_baserequest.py", line 165, in post
result = await self._request_wrapper(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/omd/sites/monitoring/local/share/checkmk-telegram-plus/telegram/request/_baserequest.py", line 330, in _request_wrapper
raise Conflict(message)
telegram.error.Conflict: Conflict: terminated by other getUpdates request; make sure that only one bot instance is running

from checkmk-telegram-plus.

Okay, but that has to do with the fact that your bot probably runs twice. Do you still have the script running somehow and at the same time the service or something like that? Have you already tried to restart the server? Maybe you still have an unintentional instance running.

from checkmk-telegram-plus.

I restarted the server. Still same...

Here is the list of running processes:
monitor+ 560 0.0 0.6 191092 28888 ? Sl 08:52 0:00 python3 /omd/sites/monitoring/bin/mkeventd
monitor+ 579 0.0 0.4 30752 16804 ? S 08:52 0:00 python3 /omd/sites/monitoring/bin/mknotifyd
monitor+ 622 0.0 2.3 154256 96496 ? S 08:52 0:01 python3 /omd/sites/monitoring/bin/cmk --create-rrd --keepalive
monitor+ 624 0.0 2.2 106268 93940 ? S 08:52 0:00 python3 /omd/sites/monitoring/bin/cmk --handle-alerts --keepalive
monitor+ 633 0.0 2.4 115260 102240 ? S 08:52 0:01 python3 /omd/sites/monitoring/bin/cmk --checker
monitor+ 634 0.0 2.3 112384 99460 ? S 08:52 0:01 python3 /omd/sites/monitoring/bin/cmk --checker
monitor+ 635 0.0 2.3 111412 98192 ? S 08:52 0:01 python3 /omd/sites/monitoring/bin/cmk --checker
monitor+ 636 0.0 2.3 111572 98296 ? S 08:52 0:01 python3 /omd/sites/monitoring/bin/cmk --checker
monitor+ 637 0.0 2.2 104596 93292 ? S 08:52 0:01 python3 /omd/sites/monitoring/bin/cmk --keepalive --real-time-checks
monitor+ 638 0.0 0.9 51396 41360 ? S 08:52 0:00 python3 /omd/sites/monitoring/bin/fetcher
monitor+ 639 0.0 0.9 51132 41468 ? S 08:52 0:00 python3 /omd/sites/monitoring/bin/fetcher
monitor+ 640 0.0 0.9 51156 41456 ? S 08:52 0:00 python3 /omd/sites/monitoring/bin/fetcher
monitor+ 641 0.0 0.9 51148 41324 ? S 08:52 0:00 python3 /omd/sites/monitoring/bin/fetcher
monitor+ 642 0.0 0.9 51156 41164 ? S 08:52 0:00 python3 /omd/sites/monitoring/bin/fetcher
monitor+ 643 0.0 0.9 51152 41432 ? S 08:52 0:00 python3 /omd/sites/monitoring/bin/fetcher
monitor+ 644 0.0 0.9 51140 41064 ? S 08:52 0:00 python3 /omd/sites/monitoring/bin/fetcher
monitor+ 645 0.0 0.9 51172 41156 ? S 08:52 0:00 python3 /omd/sites/monitoring/bin/fetcher
monitor+ 646 0.0 0.9 51252 41160 ? S 08:52 0:00 python3 /omd/sites/monitoring/bin/fetcher
monitor+ 647 0.0 0.9 51384 41376 ? S 08:52 0:00 python3 /omd/sites/monitoring/bin/fetcher
monitor+ 648 0.0 0.9 51052 40832 ? S 08:52 0:00 python3 /omd/sites/monitoring/bin/fetcher
monitor+ 649 0.0 0.9 51292 41128 ? S 08:52 0:00 python3 /omd/sites/monitoring/bin/fetcher
monitor+ 650 0.0 0.9 51140 41360 ? S 08:52 0:00 python3 /omd/sites/monitoring/bin/fetcher
monitor+ 651 0.0 1.1 57760 46320 ? S 08:52 0:00 python3 /omd/sites/monitoring/bin/cmk --notify --keepalive
root 1030 0.0 0.0 7000 3200 ? Ss 08:53 0:00 /usr/sbin/runuser -l monitoring -c cd /omd/sites/monitoring/local/share/checkmk-telegram-plus && /omd/sites/monitoring/bin/python3 /omd/sites/monitoring/local/share/checkmk-telegram-plus/telegram_bot.py
monitor+ 1047 0.0 1.2 208824 50896 ? Ssl 08:53 0:00 /omd/sites/monitoring/bin/python3 /omd/sites/monitoring/local/share/checkmk-telegram-plus/telegram_bot.py
root 1665 0.0 0.0 3240 1408 pts/3 S+ 08:54 0:00 grep python

Looks like there are two instances running - which is the correct one?

from checkmk-telegram-plus.

All are correct. So that is correct. It could also be that your API key from Telegram is in use somewhere else? Do you have 2 bots running with the same API key? That doesn't work. Can you maybe try to create a new API key, uninstall the old bot and install the new one with the new key? By itself, the whole other thing should work. Unfortunately, several people have this problem with the API key. Telegram is limited to one API key use at a time. I have also tested a few things in my lab and everything works as it should. So you must have something running twice or something.

from checkmk-telegram-plus.

This bot is dedicated for CheckMk notifications and the Key is not used somewhere else...

I can recreate a new API key in Telegram - but how do I uninstall the bot?
Do I do this also in Telegram or by uninstalling the Telegram Plus script or both?

If I remember correctly, it could be that when I ran once your install.sh, the API key was then incomplete and then I run it again with the correct API key.

Is it possible to display the connected instances and then delete/unregister them or is this not possible?

from checkmk-telegram-plus.

So to the first question: To uninstall the bot, just follow my instructions here on github. You can delete the old bot as needed.

On your other point, that could possibly be the problem. The API key is only updated during the first installation. So far I thought this is the better way etc. but since this seems to be misunderstood by users I will change it.

Here are the instructions again in detail, then you don't have to look at the documentation etc.:

1. Uninstall the Bot

omd_site_name=<omd_site_name>
systemctl stop checkmk-telegram-plus-$omd_site_name
rm /etc/systemd/system/checkmk-telegram-plus-$omd_site_name.service
systemctl daemon-reload
rm -Rf /omd/sites/$omd_site_name/local/share/check_mk/notifications/telegram_plus_notify_listener
rm -Rf /omd/sites/$omd_site_name/local/share/checkmk-telegram-plus

2. Create a new Bot / API key using the Bot Father
3. Install the Bot again

wget https://raw.githubusercontent.com/deexno/checkmk-telegram-plus/main/install.sh
bash install.sh <omd_site_name> <api_token> <bot_password>

from checkmk-telegram-plus.

all done - the menue appears again - check host and services also work - custom notification just sends mail even force and broadcast is activated...

~/local/share/checkmk-telegram-plus$ python3 telegram_bot.py shows still:
Error while getting Updates: Conflict: terminated by other getUpdates request; make sure that only one bot instance is running

from checkmk-telegram-plus.

Okay, but you stopped the service before you executed the command, didn't you? So you did:

systemctl stop checkmk-telegram-plus-<omd_site_name>
su - <omd_site_name>
cd ~/local/share/checkmk-telegram-plus
python3 telegram_bot.py

Because if you have not stopped the service, logically the error will occur again. And you have created a notification rule with the plugin in CheckMK, right? So, as it says in my documentation?

from checkmk-telegram-plus.

I repeated the test again, then rebooted the server, set up a new notification as in your documentation... Service is running fine now - but customer notification just sends mail... Telegram plus just says "I'm back" - status checks still work fine.. In time between I got 7 mails from CheckMK - but none from Telegram plus...
In addition I noticed that when I kill the telegram app on my phone and start it again -> the notifications loud or silent are deactivated again...

from checkmk-telegram-plus.

Okay, so I'm sure my bot is working as it should. I also think you have done everything correctly, but somewhere along the way something goes wrong...

Can you tell me what you get as output when you execute 🔔 LIST NOTIFY QUEUE in the Admin Settings (In the bot) and what you get when you execute 👥 LIST USERS? You can make the user IDs unrecognizable, I'm only interested in whether someone users are listed there under USERS WITH ACTIVE NOTIFICATIONS (LOUD): or not.

from checkmk-telegram-plus.

The output is:

ALLOWED USERS:
admin (xxxxxx789),

USERS WITH ACTIVE NOTIFICATIONS (LOUD):
xxxxxx789,

USERS WITH ACTIVE NOTIFICATIONS (SILENT):
xxxxxx789,

🔔 LIST NOTIFY QUEUE:
🚫 EMPTY

from checkmk-telegram-plus.

So everything looks as it should be.... And what is the output of 📖 GET LOGS? I have 3 installations running on different servers with this version, and it works without any problems...

Based on this post, you can try that?:
https://forum.checkmk.com/t/advanced-telegram-plugin-telegram-plus/37200/30?u=deexno

Did you create the notification rule exactly as I described it in the documentation? Can you send me a screenshot of it?

from checkmk-telegram-plus.

HERE ARE THE LAST 25 LOG ENTRIES::
(nothing)

in time between I rechecked the notification settings - silen and loud were again empty - 5 minutes before I copied them and they were assigned for silend and loud... so I reactivated them again and received two notifications...

Then I tried a new custom notification and it worked so far...

here is my screen shot:

So I will wait and see how it continous...

from checkmk-telegram-plus.

Okay, that's strange because the messages are only activated/deactivated by my code if you also effectively click the button in the menu... So this error is extremely strange... I would need more data to be able to understand this. Did it work now? Does the problem still exist?

from checkmk-telegram-plus.

Hi
I haven't touched the configuration since Friday and on Saturday there was an update in the Telegram app and since then all the messages come in again as they did at the beginning. Something has fixed the problem I guess. I hope that it will stay like this.
I don't understand why there were suddenly problems - everything had been working without any problems. I keep my fingers crossed and thanks for your support!

from checkmk-telegram-plus.

All right. I think you can also mess around with the config etc. as long as you understand what you are doing. Because I now have a few people who actively give me feedback on the updates and the bot, and so far I've only heard good things and I always patch problems as quickly as possible. I could also think that maybe your Telegram app had a problem or something like that? Anyway, I see the problem as solved and if you have a problem again, you can open this issue again or create a new one.

from checkmk-telegram-plus.