Comments (4)
Hey @laurentS
We are mostly using one of these tools for monitoring (push/pull)
- https://uptimerobot.com/ (Push/Pull)
- https://github.com/louislam/uptime-kuma (Push/Pull)
- Sentry (Push)
Using sentry, we have options to integrate within the codebase for centralized/easy configuration.
For e.g.:
We have sentry cron configs here https://github.com/IFRCGo/go-api/blob/59e8a463403f2cbdbcefec47ecc5c31dcadc8237/main/sentry.py#L107-L120
which are tracked automatically with just a decorator like https://github.com/IFRCGo/go-api/blob/59e8a463403f2cbdbcefec47ecc5c31dcadc8237/api/management/commands/index_and_notify.py#L60
For pull checks, /health-check endpoint is also something which seems to be good integration
For e.g.:
https://github.com/IFRCGo/alert-hub-backend/blob/75445599b6cfb17fe2749d09edad84cabde403cb/main/settings.py#L105-L111
https://alerthub-api.ifrc.org/health-check/ - Running instance
In the future, we should add more of this checks in mapswipe for identifying issues proactively.
from mapswipe.
Update:
Summary:
We have set up a simple heartbeat check that sends an "okay" message to Sentry every 2 hours. If the container stops working, the heartbeat check will not send the "okay" message, triggering an email notification to Sentry's Mapswipe members.
Details:
We have set up a cron monitor to track the status of the worker Docker container using Sentry.
We can view the monitor here: Sentry Cron Monitor
The following script is used to send the heartbeat to Sentry:
#!/bin/bash -ex
SENTRY_CRON="https://xxxxxxx.ingest.sentry.io/api/xxxxxxx/cron/workers_firebxxxxxxx/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/"
CONTAINER_NAME=mapswipe_workers_firebase_to_postgres
# 🟡 Notify Sentry your job is running:
curl "${SENTRY_CRON}?status=in_progress"
docker ps --format '{{.Names}}' | grep -wq "$CONTAINER_NAME"
sleep 2
# 🟢 Notify Sentry your job has completed successfully:
curl "${SENTRY_CRON}?status=ok"
The crontab configuration to run this script every 1 hours is as follows:
0 */1 * * * /root/sentry-cron-monitor.sh
Future Improvements
To enhance our system's reliability and ensure timely execution of scheduled tasks, we should implement cron monitoring for each individual task using the Sentry Python SDK. This will allow us to detect and address any issues with task execution promptly.
from mapswipe.
For future improvements, we have created a ticket for in-depth tasks monitoring
from mapswipe.
This is a great step forward! Out of curiosity (I'm researching monitoring options for similar use cases) did you look at other options?
from mapswipe.
Related Issues (20)
- TM Statistics problems HOT 1
- investigate/ adjust sentry logging
- Add option to tap to go to next page
- Explore the possibility of adding a banner in the app HOT 1
- RTL languages break app
- duplication of usernames
- project info modal should handle overflow text with scroll
- Null island point in 'HOT Tasking Manager Geometries'
- MapSwipe app crashing in certain projects HOT 4
- Edit group names after being created HOT 3
- Group contributor export download not complete
- MapSwipe Dev New Tutorial Setup Timeout HOT 1
- MapSwipe Dev Tutorial Imagery Inconsistency between Web setup and App Tutorial HOT 3
- Test
- Too difficult to swipe up/down HOT 3
- Add qraphql unique ID for caching data policy
- MapSwipe password reset issue HOT 1
- Community statsboard not working HOT 1
- MapSwipe Dev Tasks not showing up in api index - for .csv.gz file use HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mapswipe.