humanmade / cavalcade Goto Github PK
View Code? Open in Web Editor NEWA better wp-cron. Horizontally scalable, works perfectly with multisite.
Home Page: https://engineering.hmn.md/projects/cavalcade/
License: Other
A better wp-cron. Horizontally scalable, works perfectly with multisite.
Home Page: https://engineering.hmn.md/projects/cavalcade/
License: Other
I am running a version of Altis+Chassis+Wordpress. There is an issue with the Cavalcade module: it throws errors when I have the dubugger open. See screenshot.
I am assuming Cavalcade is a dependency of one of the Altis modules. How do I fix this and confirm that Cavalcade is doing whatever it is supposed to do?
This is the Altis part of my composer.json file:
"require-dev": {
"altis/core": "dev-master",
"altis/local-chassis": "dev-master",
"altis/dev-tools": "dev-master",
"humanmade/coding-standards": "^0.7.0"
},
"autoload-dev": {
"files": []
},
"extra": {
"installer-paths": {
"content/plugins/{$name}/": [
"type:wordpress-plugin"
],
"content/mu-plugins/{$name}": [
"type:wordpress-muplugin"
],
"content/themes/{$name}/": [
"type:wordpress-theme"
]
},
"altis": {
"modules": {
"local-chassis": {
"hosts": [
"fresenius.local"
],
"multisite": "Yes",
"extensions": [
"sequelpro",
"xdebug",
"memcache",
"phpini"
]
}
}
}
}
On our install, sometimes we see the same job for the same hook being duplicated multiple times with the same nextrun
timestamp.
Regardless of my example, instead of using a SQL query to delete a job or multiple jobs, it would be handy if there was a wp-cli command that allowed us to remove said jobs.
If this sounds like a good idea, I recommend wp cavalcade job delete
for this new subcommand, but that means a wp cavalcade job
command should be created as this currently doesn't exist.
What do you think?
Hi,
We are using zero downtime deployments for our project (Deployer). When deploying some new code the worker seems to run into problems. All jobs continuously fail with the message "Failed to shutdown worker.". I figure this has something to do with with paths/symlink changing when a new deployment is executed.
The only fix I found for now is to stop the supervisor job completely en restart it again, I was wondering if this is a known problem and if there would be a more low level solution. As restarting supervisor requires root and is therefor a pain to integrate in our deployment logic.
Thanks in advance.
Cavalcade prevents the rescheduling of an existing job to a new frequency as the change in scheduling is not recognised.
Including the interval in the connector's real key will resolve the problem.
To reproduce the error, run the following:
$stamp = time() + HOUR_IN_SECONDS;
wp_schedule_event( $stamp, 'hourly', 'cavalcade_repeat' );
wp_schedule_event( $stamp, 'twicedaily', 'cavalcade_repeat' );
var_dump( _get_cron_array()[ wp_next_scheduled( 'cavalcade_repeat' ) ] );
The final line will show the job is scheduled to run hourly.
Jobs containing args that contain multibyte unicode characters such as emoji can't be run as after they're serialised they do not unserialise correctly.
There's a chance that you might end up with a stack of jobs running at once which can throttle a server. We should consider adding a command to stop jobs from running.
@rmccue does that sound like a good idea? Can you forsee any issues or reasons not to add a command for that?
After what I can only describe as an intense session of debugging with @rmccue we discovered that some jobs were hanging when the pipe's buffer filled up from stderr / stdout of a running job. This could be something like a cron job doing an excessive amount of error_logging.
The buffer size is relatively small by default, and I believe the runner only reads from the pipe once the child process has completed. This means the sys call to write to the pipe will hang when full.
As a "workaround" we removed the error_log
calls from the job, and if need be it's possible to cat /proc/{process id of wp-cli}/fd/2
to flush the pipe buffer.
Since out of the box monitoring is missing it would be nice to have the possibility of automatically retrying failed jobs. Along with a timeout this would have prevented almost all issues we've had.
Hey, this is Luca from FFX. Our paths cross again =)
I can see both CAVALCADE_JOB_ID
and DOING_CRON
constants are defined in Command::run() just before the actual execution of the scheduled hook.
That means I cannot access them during, say, any init
or admin_init
hook.
I'm trying to switch to Cavalcade for our multisite, but every time I start runner, it crashes after running the first set of jobs with the following error:
Sep 15 14:28:18 server php: PHP Notice: Undefined offset: 0 in /opt/Cavalcade-Runner/lib/Runner.php on line 211
Sep 15 14:28:18 server php: PHP Fatal error: Call to a member function is_done() on a non-object in /opt/Cavalcade-Runner/lib/Runner.php on line 212
I've been running it in the foreground, but it doesn't really output anything more useful:
cavalcade
[27] Running wp cavalcade run 27 --url='f.q.d.n/site1/' (check_plugin_updates-user-role-editor-pro a:0:{})
[27] Started worker
[27] out:
[27] err:
[27] ret: 0
[28] Running wp cavalcade run 28 --url='f.q.d.n/site2/' (check_plugin_updates-user-role-editor-pro a:0:{})
[28] Started worker
[29] Running wp cavalcade run 29 --url='f.q.d.n/site2/' (wp_scheduled_delete a:0:{})
[29] Started worker
Actual hostnames and URLs replaced to protect the innocent, etc. Meanwhile, I'm also looking into why the stupid User Role Editor Pro checks for plugin updates on every single site.
The cron array may be accessed prior to the plugins_loaded
action firing, the most likely candidates are:
muplugins_loaded
plugin_loaded
added in WP 5.1muplugin_loaded
added in WP 5.1Running Connector\bootstrap()
as the plugin is require
d will filter the cron array as soon as the code is available.
On the downside, the is_installed()
check would probably need to be included in the filters but as it's cached, this shouldn't cause too much of a problem.
Follow up to discussion in PR #91
Cavalcade uses a non-persistent cache for the cavalcade-jobs
group. As the runner triggers WP CLI tasks within the plugin, invalidating on UPDATE, INSERT and DELETE should be a relatively simple task.
Proposal:
cavalcade-jobs
from the non-persistent groupscavalcade-jobs
should be a global or per siteHello,
My question is about current big installions that will migrate to WP native cron to Calvalcade cron system.
Is it possible ? I think we can
but is it safe ?
Regards,
The WP-CLI commands have sparse user-facing documentation, which does not give the user everything they need to know.
For example, the @synopsis
tag tells me that the jobs
command has a status
filter, but not what the valid values for it are. I have to dig through the code to figure out that they're -- as far as I can tell -- waiting
, running
, failed
, and completed
.
Adding option descriptions and examples would result in a much better UX.
Hey, so we're using a WP multisite setup, and noticed that when a site is deleted, Cavalcade doesn't clean out the jobs for these non-existent sites.
add_action('delete_blog', 'cleanup')
function cleanup($site_id) {
// remove jobs owned by $site_id
}
Parse error: syntax error, unexpected '[', expecting ')' in /opt/wordpress/wp-content/plugins/Cavalcade-master/inc/namespace.php on line 22
I am using one long process which takes about an hour to get completed. but the process get in failed status before it get completed.
Can your docs please mention that pcntl must be installed?
http://php.net/manual/en/book.pcntl.php
I just noticed that DOING_CRON isn't defined when running jobs, this could cause unexpected behaviour for some jobs.
While running through WP-CLI, it won't be possible to define it before bootstrap occurs (AFAIK), but defining it late might be better than not at all.
I dropped all 4 of the files into the mu-plugins folder
class-command.php
class-job.php
connector.php
plugin.php
And on reload immediately the site died with a 500 error
PHP Fatal error: Class 'WP_CLI_Command' not found in wp-content/mu-plugins/class-command.php on line 8
Can you please update or add detail to your install instructions?
Last evening, at exactly 9 PM EDT (2 AM GMT), hourly jobs started running continuously--instead of being scheduled for the next hour. This caused a major load event and we had to stop Cavalcade everywhere. Sample log showing the issue:
20413 wp_privacy_delete_old_export_files 2018-11-03 19:14:32 completed
20413 wp_privacy_delete_old_export_files 2018-11-03 20:14:32 completed
20413 wp_privacy_delete_old_export_files 2018-11-03 21:14:34 completed
20413 wp_privacy_delete_old_export_files 2018-11-03 21:14:36 completed
20413 wp_privacy_delete_old_export_files 2018-11-03 21:14:37 completed
20413 wp_privacy_delete_old_export_files 2018-11-03 21:14:39 completed
20413 wp_privacy_delete_old_export_files 2018-11-03 21:14:40 completed
20413 wp_privacy_delete_old_export_files 2018-11-03 21:14:41 completed
20413 wp_privacy_delete_old_export_files 2018-11-03 21:14:42 completed
This morning, it seems like we can get cavalcade happy by running wp site list --field=url | xargs -n 1 -I % wp cron event run --due-now --url=%
to bring all jobs into the present. The WordPress site in question uses UTC internally; the host servers (both application and database) are on EDT/EST.
Running wp cavalcade jobs --status=completed
will result in the command fetching potentially millions of rows, which takes forever (if it even finishes) and doesn't provide meaningful results to the user.
A default limit of 10
or 20
and a --limit
parameter seems like a good solution.
Ran into an interesting scenario today where a daily recurring job failed last week and it wasn't queued again / re-run.
In the cavalcade_jobs
table, the status
was 'failed'
wp_next_scheduled( $job )
returned the date for the next time after that failed run when it should have run (The plugin queries for jobs which are status != 'complete'
to return in the cron array)status = 'waiting'
.This ended up in a situation where WordPress thought the job was queued and Cavalcade didn't attempt to run it.
In this case, I manually reset the status
field for the jobs in question to waiting
and it continued on it's way.
Currently get_next_job()
does two things wrong IMHO:
LIMIT 1
- which while implied by the fetchObject()
call, still causes all waiting jobs to be returned to the client bufferORDER BY
- What this causes is that when jobs get backlogged, jobs run in an order as determined by their id
rather than by their order of nextrun
, which can result in some unexpected behaviours in cron jobs (if a specific time, say an hour after another job was specified, and can cause some unexpected behaviours in cron jobs)The ORDER BY
shouldn't be an issue, except when backlogged by something such as #37
Similar to #18, we ran into an issue on WordPress.org where one of cavalcade daemons was killed unexpectantly which left a bunch of jobs in an unknown state.
The jobs were marked as running, but there were no workers to manage those jobs anymore. This resulted in jobs not running for a few hours/days until they were detected and restarted.
There should be some way for a job to be detected as no-longer-running or that it's daemon is MIA.
In this case, I simply restarted the jobs: UPDATE wp_cavalcade_jobs SET status = 'waiting' WHERE status = 'running' AND nextrun <= '2016-12-06'
@rmccue Cavalcade looks great and attacks a real problem with wp-cron and larger multisites. Is this ready to give a try with the expectations of a few minor issues but nothing crazy?
I'd be happy to give our feedback on trying to use this if you need some beta testers to help the project.
Use case is an event which must come after the completion of another event.
example:
Event_A
calls wp_schedule_single_event( time(), 'event_b' );
Event_A
shows up as completed in the log but event_ b
never shows up in the log (neither the DB log or a piped to log from the cavalcade runner).
It seems that perhaps the Cavalcade WP plugin doesn't support adding a job using a class with an invokable as the callback?
http://php.net/manual/en/language.oop5.magic.php#object.invoke
I have the following code to add my invokable class:
if ( ! wp_next_scheduled( Model\MyExampleInvokableClass::class ) ) {
wp_schedule_event( time(), 'daily', Model\MyExampleInvokableClass::class );
}
This seems to work fine using the standard WP Cron system and I'm able to verify that the job is queued using the 'Advanced Cron Manager' plugin (https://wordpress.org/plugins/advanced-cron-manager/).
However, when Cavalcade is enabled, these jobs disappear from the jobs list in WP Admin and also aren't visible when I do wp cavalcade jobs
.
I don't see anything in PHP error logs or cavalcade log either...
Thanks! ๐
Running composer install
will also check for php extensions and versions, adding these to composer.json
would help with checking dependencies
Currently the only way to get job failure data is to look at the main cavalcade output logs, this is useful when Cavalcade is only running on a single server, or a bunch of servers with a remote syslog, but when you need to find the failure data for a job that was run on one of many different servers it requires digging through multiple server logs to find the actual data.
It'd be ideal if instead of/in addition to logging Failed to shutdown worker.
that the stdout
and stderr
streams were logged to the database.
WordPress.org currently runs Cavalcade with a modified Runner::check_workers()
so that we log the server on which the job ran, we could add this functionality ourselves but Job->output
and Job->error_output
are both protected
fields which prevents this.
Unfortunately there isn't an easy way to override the Worker
class so we can't simply extend it as we've done with the Runner
(And if we wanted to put stderr
/stdout
into their own field in the table, rather than a blob of data into content
we'd also need to override Logger
which again, isn't easily possible).
I'm setting up Cavalcade in a hosting environment where MySQL is running in local time - UTC-5, in my case. This is causing a mismatch: WP and mu-plugins/cavalcade are recording the job (properly) in UTC, while the runner's nextrun < NOW()
query is offset by 5 hours, causing jobs to be run 5 hours late.
I realize that the ideal situation is to use UTC everywhere, but I don't control the environment, and certain parts of the infrastructure are shared with a couple other resources.
I've fixed this in my local codebase by putting some offset logic in the /connector/namespace.php
methods. I'm wondering whether you'd be interested in an enhancement that makes it possible to set a flag (a constant or something) so that Cavalcade will perform this offset automatically if the flag is set.
Feel free to close if you feel like this would be reinforcing a bad practice! And thanks for the cool tool ๐
In WordPress/wordpress-develop@e73af26 meaningful return values were introduced for cron functions, often these return the result of update_option()
for the cron array.
As the Cavalcade connector short circuits the cron array from updating this always returns a false
. Until the connector is updated to use the hooks added in WordPress/wordpress-develop@a32ea2d testing the return values will fail.
Blame is an ugly word but it should be noted both of the commits linked above were done by the same person. Me.
Occasionally, jobs may get stuck. Having a high limit of something like 30 mins (configurable) would help catch these.
Adding a cron job with the code wp_schedule_event( 0, 'daily', 'cavalcade_daily' );
will cause the job to run constantly until it's run once for each day since 1/1/1970. Once the job has caught up it will run at midnight each day, as expected.
Adding a command such as wp cavalcade job restart {id} --skip-backlog
will allow users to skip the backlog and ensure the job only runs once per day.
Follow up to humanmade/Cavalcade-Runner#53
Can you provide step to install you plugin ?
I get error when install on mu plugin
PHP Fatal error: Class 'WP_CLI_Command' not found in /var/www/wp-content/mu-plugins/class-command.php on line 8
I can run wp --info
PHP binary: /usr/bin/php5
PHP version: 5.5.9-1ubuntu4.7
php.ini used: /etc/php5/cli/php.ini
WP-CLI root dir: phar://wp-cli.phar
WP-CLI global config:
WP-CLI project config:
WP-CLI version: 0.20.1
what next should I do?
thank you
Currently cavalcade doesn't support wp_get_schedule()
properly - it always returns __fake_schedule
. This isnt' a problem for WordPress normally, however it has caused issues with the latest Jetpack release.
Consider the following code, boiled down from https://plugins.trac.wordpress.org/browser/jetpack/tags/4.4/sync/class.jetpack-sync-actions.php?marks=295-302#L289
if ( ! wp_next_scheduled( $hook ) ) {
wp_schedule_event( time(), 'twicedaily', $hook );
} else if ( 'twicedaily' != wp_get_schedule( $hook ) ) { // It was queued as 'daily' previously
wp_clear_scheduled_hook( $hook );
wp_schedule_event( time(), 'twicedaily', $hook );
}
Cavalcade will return __fake_schedule
for wp_get_schedule()
and cause it to endlessly clear the hook and requeue it again.
As we've run into this on WordPress.org, I've temporarily put the following in place: dd32@fbd23d2
Unfortunately as pointed out in the commit, this is only a temporary solution, in the event that two schedules with identical timings, but different names, there's a good chance that it'll return the incorrect schedule.
I don't really see a way around this other than storing the schedule name in the database which a job is attached to, that probably means an extra field in the jobs table.
Hey,
Not quite sure where to start debugging this tbh, but we've been having issues of tasks not getting cleared properly. We've also had a few times where we had to reenable plugins to make scheduled tasks run.
Feel free to push me in the right direction for getting some useful information that can be used for debugging this.
Some info from the developer of Product Bundles:
I can see that tasks are completing, which rules out a problem with
WC_PB_DB_Sync_Task_Runner::time_exceeded or
WC_PB_DB_Sync_Task_Runner::memory_exceeded.
So WC_PB_DB_Sync_Task_Runner::task is returning false, which is correct.
However, there's likely a problem here: https://cl.ly/3q293d3y1s1Z
Tasks look like they are not being cleared out of the options table correctly
That's something you need to look into because the WC updater relies on the same library -- if it doesn't work correctly, you'll likely have issues with updating WooCommerce down the road.
Can you perhaps debug this on a standard / local server environment without any complex caching or task scheduling software running?
You should see this instead of the same task being processed over and over:
2017-10-31T07:48:52+00:00 INFO Sync complete.
If this is a problem that affects your live site only, it might be hard to diagnose it -- and something my support team won't be able to help with.
I am not very familiar with Cavalcade either -- if it's a caching or Cavalcade related issue, I'd have trouble figuring out the exact cause without looking at their code. Something like that could take days to debug :)
Can you share your findings after looking into it on the staging site?
Keep an eye on that log -- it should not keep running over and over with every request.
Tasks are stored in your options table -- they should be called:
wp_wc_pb_db_sync_task_runner_batch_xxx
If you have a lot of these queued, it might take some time until they get processed, but eventually you should see a
Hi!
We are using Cavalcade for a crucial part of our system. Can we be 100% certain that failed jobs will remain in the cavalcade_jobs
table so we can re-run them at a later point in time, or is there any chance that they might disappear?
Thanks!
Hi Ryan/Humanmade
First of all thanks for creating this, have been looking for something like this for a while to have better job/queue management.
Coming from Laravel I have the feature request to run tasks sync on local/staging environments for easier development/debugging.
An other workaround suggestion would also be appreciated!
Thanks!
When I try to run wp cavalcade log
on with wp-cli 1.5.1 and wp-cli 1.5.0, I'm getting this error:
https://5e8b25de-f7a8-4b18-934e-9f3b0d54042e.htmlpasta.com/
It works fine with wp cli 1.4.1 without any errors.
Any ideas ?
The Cavalcade tables can be created prior to WP Core tables.
A couple of times I have seen the Cavalcade tables created in the database before the options table exists. Since Cavalcade started storing it's own DB version in the options table, this can now cause problems.
Prior to bootstrapping, a check to ensure the options table is created and populated will prevent this. For example:
if ( ! get_option( 'db_version' ) ) {
return;
}
A ready filter would be handy that runs the pseudo SQL query SELECT jobs WHERE status='waiting' AND 'nextrun' < time()
.
When attempting to install WP on my wp-local-docker environment, I get the following error:
WordPress database error Table 'wordpress.wp_options' doesn't exist for query INSERT INTO `wp_options` (`option_name`, `option_value`, `autoload`) VALUES ('cavalcade_db_version', '2', 'no') ON DUPLICATE KEY UPDATE `option_name` = VALUES(`option_name`), `option_value` = VALUES(`option_value`), `autoload` = VALUES(`autoload`) made by include('phar:///usr/local/bin/wp/php/boot-phar.php'), include('phar:///usr/local/bin/wp/php/wp-cli.php'), WP_CLI\bootstrap, WP_CLI\Bootstrap\LaunchRunner->process, WP_CLI\Runner->start, WP_CLI\Runner->load_wordpress, require('wp-settings.php'), do_action('plugins_loaded'), WP_Hook->do_action, WP_Hook->apply_filters, HM\Cavalcade\Plugin\bootstrap, HM\Cavalcade\Plugin\create_tables, update_site_option, update_network_option, add_network_option, add_option
The problem appears to be line 98 in inc/bootstrap.php
, within the create_tables
function:
update_site_option( 'cavalcade_db_version', DATABASE_VERSION );
The error is being caused by exactly what it says on the tin -- the wp_options
table is not in existence at the time that this function call is made. It's possible that this is something wrong with my docker environment, but I note that no other plugins cause issues.
System details:
Task
When setting and getting Cron jobs, make use of the new WP filters: pre_schedule_event
, pre_reschedule_event
, pre_unschedule_event
, pre_clear_scheduled_hook
, pre_unschedule_hook
, pre_get_scheduled_event
and pre_get_ready_cron_jobs
.
Background
In WordPress 5.1, new filters will be added to preflight event scheduling to allow for custom Cron storage systems.
The filters were added in WordPress#32656 (see WordPress/wordpress-develop@a32ea2d) and WordPress#45797 (see WordPress/wordpress-develop@0bc2059).
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.