Comments (5)
A daemon being killed off while running jobs is especially relevant in horizontally scaled setups. If a node is taken offline, then the jobs can get stuck in this state, and have to be restarted manually.
from cavalcade.
@larssn Indeed, we've been working on solving this ourselves. It's tough to come up with a solution to it. Mostly, we've been focusing on ensuring the daemon safely shuts down the workers.
If anyone has better ideas, all ears. :)
from cavalcade.
Could you make use of the flock
side-effect of clearing file locks when the PHP process dies? We've used this on BackUpWordPress (xibodevelopment/backupwordpress#1025) to detect when a long running process has crashed / is killed so we can update it's status accordingly rather than having it forever show as (incorrectly) running.
from cavalcade.
Unfortunately not, due to the horizontal distribution. IIRC NFS doesn't support file locks either.
from cavalcade.
The "correct" approach here for horizontally distributed apps would be for each daemon to have a DB row listing it's status, and where it's running from, with a date stamp bumped every minute or so.
Jobs would then have to be listed as status = running, server = ID#4
.
Other daemons would need to periodically check the table to see if any other daemons had started, accepted jobs, and gone away without marking themselves as shutdown, and initiate a cleanup.
It'd probably need to be implemented as a wp-cli job which each daemon fires off every ~5mins (ie. it can't be a listed job, it'd have to be a custom thing, as you want it on each server) which performs the health checks.
from cavalcade.
Related Issues (20)
- Cron filters are not registered early enough HOT 3
- --allow-root HOT 3
- Access to wp-cron.php should be disabled when Cavalcade is in use HOT 2
- Issues with the index on `nextrun` HOT 14
- PHP 8.0 composer support
- WordPress 5.7 support HOT 3
- wp_schedule_single_event not executing function when calvacade is running on server but works locally
- Results of get_jobs_by_query can give stale timestamps for recurring jobs
- Existing jobs HOT 2
- Some CRON events are not executed (Large WP Multisite) HOT 10
- Question: How do determine whether Cavalcade is active from other Code? HOT 1
- Cavalcade is not working? HOT 1
- Why does Cavalcade do so much on every page load? HOT 3
- Performance issues if checking for schedules during every page load HOT 11
- Command that takes a job id and sets status to 'failed'
- Quick questions HOT 2
- Possible Wrong Version Display and LOG disabling solution HOT 2
- PHP Warning with PHP8.2 HOT 2
- Cannot unschedule more than 100 events with a given hook
- Allow to not replay every missing events HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cavalcade.