ostark / craft-async-queue Goto Github PK

I'm running an older version of Craft and PHP 7.1 and using this craft-async-queue 1.3.3.

I see in the docs that you mention that by setting the PHP_BINARY var in your .env you can update your PHP binary location. I have done this but I still see an error related to this plugin:

Unable to find php binary.

Is there anything else I need to do to tell Async Queue where to look for the binary after adding the var to .env?

First of all, thank you for writing this great plugin. In this case, I need craft-async-queue because of the way Craft Commerce uses the Queue to send Status event emails, such as when an Order is placed. The problem I am having is that when you attach a PDF (which is a Craft generated invoice) to the email, craft-async-queue does it's job and sends out the email promptly, however, when craft-async-queue is handling the queue, the PDF attachment is always corrupted. It provides a blank PDF file with the single line contained:

An error occurred while processing the PDF.

When I disable craft-async-queue, the PDF gets emailed successfully. This is completely reproducible by just turning the plugin on and off. On: almost immediately sent email but corrupted PDF. Off: good PDF, but painfully slow email send by having to click around in the admin to initializing the queue job with Craft's native methods.

I tried digging through storage/logs/queue.log and storage/logs/web.log and phperrors, and even through all the system log files in /var/log to try to trace down where this is failing but there is no record of any failure in any of the logs. I tried grepping pdf in all of them and nadda.

I know that Craft Commerce uses dompdf behind the scenes to perform the PDF generation. I suspect that the daemon that craft-async-queue sets up is having either tmp file creation, or permission based or relative path issues in being able to successfully generate the PDF but I cannot find any clue to where the error is actually occurring.

I thought I would open an issue in case you could help me figure out where the problem might be occurring and in case others either have/had this same problem and have solved it or are in the same boat.

In case it helps, I'm using Craft Pro 3.5.12, Craft Commerce Pro 3.2.7, AsyncQueue 2.1.1 on stock Ubuntu 20.04.1 LTS running nginx/1.18.0 and PHP 7.4.3 with php-fpm. Dompdf version that is installed is 0.8.6

Help?

Hi, thanks for the great work in this plugin.

However, after installing it in an Ubuntu system (Forge provisioned) with the latest Craft 3 release, there's just no daemon running..?

Is there other setup needed? Do I need to disable automatic task running myself?

Here's an output of ps auxf | grep php:

forge     5504  0.0  0.0  14856   944 pts/0    S+   12:21   0:00  |           \_ grep --color=auto php
root      3832  0.0  0.2 1383384 40552 ?       Ss   11:08   0:00 php-fpm: master process (/etc/php/7.2/fpm/php-fpm.conf)
forge     5375  4.7  0.5 1484660 90000 ?       S    12:08   0:35  \_ php-fpm: pool www
forge     5376  4.7  0.5 1478548 84740 ?       S    12:08   0:35  \_ php-fpm: pool www
forge     5446  4.9  0.5 1479788 85464 ?       S    12:09   0:35  \_ php-fpm: pool www

The above was captured during a 'deleting stale template caches' task, so there was at least several tasks in the queue - it seems like php-fpm is still doing the work.

web.log contains no instances of craft-async-queue, phperrors.log lists nothing.

Any ideas?

I've been using this plugin for a while now and I love it, thank you for developing it!

One of my Craft sites is hosted in Azure with autoscaling enabled i.e. Azure will spin up multiple instances of the web app with the same configuration, pointing to the same db and Redis cluster. I am not currently using Redis for the queue, and I still have the queue running automatically (all Craft defaults). However I am running craft queue/run on a 5 minute cron to catch any instances where the queue might be jammed.

First question: is this cron necessary? Or is this only needed if runQueueAutomatically is disabled? Can I safely remove it? I do sometimes see the queue stopped, with items sitting in it, and the only way I can manually flush is to run the Async Queue Test which gets it going again.

Second: what will happen when my app is scaled? Obviously this means more worker processes since the queue is running in each instance. Does Craft and this plugin do locking to make sure items on the queue aren't grabbed by more than one worked? Are these locks based on the file system (and therefore only work in a single instance) or at the db/Redis level?

Many thanks!

Hi there,

I'm typically using your plugin with ImageOptimize but I deactivated it so I can isolate the issue.

I have a fairly simple setup, using AWS S3 bucket with CloudFront for all my assets.
Without your plugin, going to the CP and updating asset indexes works fine.
With your plugin installed, doing the same takes way more time and generates errors both in console.log and queue.log:
User 'xxx' has exceeded the 'max_user_connections' resource (current value: 10)'.
This is on a staging environment on fortrabbit FWIW.

Concurrency is set to 1.
Craft 4.0.3 / Plugin 3.0

Hi

We've just installed the AsyncQueue plugin. It works fine locally, but on our staging server Commerce emails are not being added to the queue. The AsyncQueue test does add jobs to the queue, but they don't get processed.

There's nothing in the logs. The queue log is showing no new activity since the plugin was installed.

My initial thought is that it might be a permissions issue. Could you elaborate on the "Permissions to execute a php binary" requirement? How would I check if my server has the necessary permissions enabled?

Craft 3.6.6
AsyncQueue 2.2.0
PHP 7.4

Thanks

After reading this thread, which led me to NYStudio107's #1 solution to robust queue handling, this plugin sounds like it could help with my slow Feed Me import :)

But when I go to install it via Craft 3.4.18's Plugin Store I get a verbose error message, the top portion of which is:

Composer output: Package "craftcms/vue-asset" listed for update is not installed. Ignoring.
Package "danielstjules/stringy" listed for update is not installed. Ignoring.
Loading composer repositories with package information
Updating dependencies
Your requirements could not be resolved to an installable set of packages.

Problem 1
- symfony/process v4.2.0 requires php ^7.1.3 -> your PHP version (7.2.29) overridden by "config.platform.php" version (7.0) does not satisfy that requirement.

There then follows a string of similar but not identical symfony/process messages, and then:

- Installation request for ostark/craft-async-queue 2.1.1 -> satisfiable by ostark/craft-async-queue[2.1.1].

Running update with --no-dev does not mean require-dev is ignored, it just means the packages will not be installed. If dev requirements are blocking the update you have to resolve those problems.

Hey @ostark 👋,

Plugin version: 2.1.0

Not sure if this a bug, but thought I'd report it just in case. I hit an error when attempting to upgrate a site that uses async-queue from Craft 3.3.20.1 => 3.4.11.

When running ./craft migrate/all I got this error (full stack trace after):

Exception: SQLSTATE[42S22]: Column not found: 1054 Unknown column 'channel' in 'where clause'
The SQL being executed was: UPDATE `queue` SET `dateReserved`=NULL, `timeUpdated`=NULL, `progress`=0, `progressLabel`=NULL WHERE ((`channel`='queue') AND (`fail`=FALSE)) AND (`timeUpdated` < 1585647734 - `ttr`) (/app/vendor/yiisoft/yii2/db/Schema.php:674)

Disabling async-queue and running again (albeit with a fresh database dump) worked, so it's not blocking me any more, just thought you should know about it.

Happy to try and make a reduced test case if that's helpful....

Stack trace:

The SQL being executed was: UPDATE `queue` SET `dateReserved`=NULL, `timeUpdated`=NULL, `progress`=0, `progressLabel`=NULL WHERE ((`channel`='queue') AND (`fail`=FALSE)) AND (`timeUpdated` < 1585647734 - `ttr`) (/app/vendor/yiisoft/yii2/db/Schema.php:674)
#0 /app/vendor/yiisoft/yii2/db/Command.php(1298): yii\db\Schema->convertException(Object(PDOException), 'UPDATE `queue` ...')
#1 /app/vendor/yiisoft/yii2/db/Command.php(1093): yii\db\Command->internalExecute('UPDATE `queue` ...')
#2 /app/vendor/craftcms/cms/src/queue/Queue.php(653): yii\db\Command->execute()
#3 /app/vendor/craftcms/cms/src/queue/Queue.php(341): craft\queue\Queue->_moveExpired()
#4 /app/vendor/ostark/craft-async-queue/src/RateLimiter.php(39): craft\queue\Queue->getTotalReserved()
#5 /app/vendor/ostark/craft-async-queue/src/Handlers/BackgroundQueueHandler.php(36): ostark\AsyncQueue\RateLimiter->canIUse('Resaving elemen...')
#6 [internal function]: ostark\AsyncQueue\Handlers\BackgroundQueueHandler->__invoke(Object(yii\queue\PushEvent))
#7 /app/vendor/yiisoft/yii2/base/Event.php(312): call_user_func(Object(ostark\AsyncQueue\Handlers\BackgroundQueueHandler), Object(yii\queue\PushEvent))
#8 /app/vendor/yiisoft/yii2/base/Component.php(636): yii\base\Event::trigger('craft\\queue\\Que...', 'afterPush', Object(yii\queue\PushEvent))
#9 /app/vendor/yiisoft/yii2-queue/src/Queue.php(197): yii\base\Component->trigger('afterPush', Object(yii\queue\PushEvent))
#10 /app/vendor/craftcms/cms/src/queue/Queue.php(174): yii\queue\Queue->push(Object(craft\queue\jobs\ResaveElements))
#11 /app/vendor/verbb/navigation/src/services/Nodes.php(150): craft\queue\Queue->push(Object(craft\queue\jobs\ResaveElements))
#12 [internal function]: verbb\navigation\services\Nodes->afterSaveSiteHandler(Object(craft\events\SiteEvent))
#13 /app/vendor/yiisoft/yii2/base/Event.php(312): call_user_func(Array, Object(craft\events\SiteEvent))
#14 /app/vendor/yiisoft/yii2/base/Component.php(636): yii\base\Event::trigger('craft\\services\\...', 'afterSaveSite', Object(craft\events\SiteEvent))
#15 /app/vendor/craftcms/cms/src/services/Sites.php(795): yii\base\Component->trigger('afterSaveSite', Object(craft\events\SiteEvent))
#16 /app/vendor/craftcms/cms/src/services/ProjectConfig.php(1099): craft\services\Sites->handleChangedSite(Object(craft\events\ConfigEvent))
#17 [internal function]: craft\services\ProjectConfig->handleChangeEvent(Object(craft\events\ConfigEvent))
#18 /app/vendor/yiisoft/yii2/base/Component.php(627): call_user_func(Array, Object(craft\events\ConfigEvent))
#19 /app/vendor/craftcms/cms/src/services/ProjectConfig.php(675): yii\base\Component->trigger('updateItem', Object(craft\events\ConfigEvent))
#20 /app/vendor/craftcms/cms/src/helpers/ProjectConfig.php(89): craft\services\ProjectConfig->processConfigChanges('sites.1fae7cf9-...')
#21 /app/vendor/craftcms/cms/src/services/Sections.php(595): craft\helpers\ProjectConfig::ensureAllSitesProcessed()
#22 /app/vendor/craftcms/cms/src/services/ProjectConfig.php(1099): craft\services\Sections->handleChangedSection(Object(craft\events\ConfigEvent))
#23 [internal function]: craft\services\ProjectConfig->handleChangeEvent(Object(craft\events\ConfigEvent))
#24 /app/vendor/yiisoft/yii2/base/Component.php(627): call_user_func(Array, Object(craft\events\ConfigEvent))
#25 /app/vendor/craftcms/cms/src/services/ProjectConfig.php(675): yii\base\Component->trigger('updateItem', Object(craft\events\ConfigEvent))
#26 /app/vendor/craftcms/cms/src/services/ProjectConfig.php(1089): craft\services\ProjectConfig->processConfigChanges('sections.0b8ab5...')
#27 [internal function]: craft\services\ProjectConfig->handleChangeEvent(Object(craft\events\ConfigEvent))
#28 /app/vendor/yiisoft/yii2/base/Component.php(627): call_user_func(Array, Object(craft\events\ConfigEvent))
#29 /app/vendor/craftcms/cms/src/services/ProjectConfig.php(672): yii\base\Component->trigger('addItem', Object(craft\events\ConfigEvent))
#30 /app/vendor/craftcms/cms/src/services/ProjectConfig.php(489): craft\services\ProjectConfig->processConfigChanges('sections.0b8ab5...', true, '')
#31 /app/vendor/craftcms/cms/src/migrations/m190913_152146_update_preview_targets.php(54): craft\services\ProjectConfig->set('sections.0b8ab5...', Array)
#32 /app/vendor/craftcms/cms/src/db/Migration.php(52): craft\migrations\m190913_152146_update_preview_targets->safeUp()
#33 /app/vendor/craftcms/cms/src/db/MigrationManager.php(233): craft\db\Migration->up(true)
#34 /app/vendor/craftcms/cms/src/db/MigrationManager.php(153): craft\db\MigrationManager->migrateUp(Object(craft\migrations\m190913_152146_update_preview_targets))
#35 /app/vendor/craftcms/cms/src/services/Updates.php(225): craft\db\MigrationManager->up()
#36 /app/vendor/craftcms/cms/src/console/controllers/MigrateController.php(259): craft\services\Updates->runMigrations(Array)
#37 [internal function]: craft\console\controllers\MigrateController->actionAll()
#38 /app/vendor/yiisoft/yii2/base/InlineAction.php(57): call_user_func_array(Array, Array)
#39 /app/vendor/yiisoft/yii2/base/Controller.php(157): yii\base\InlineAction->runWithParams(Array)
#40 /app/vendor/yiisoft/yii2/console/Controller.php(164): yii\base\Controller->runAction('all', Array)
#41 /app/vendor/yiisoft/yii2/base/Module.php(528): yii\console\Controller->runAction('all', Array)
#42 /app/vendor/yiisoft/yii2/console/Application.php(180): yii\base\Module->runAction('migrate/all', Array)
#43 /app/vendor/craftcms/cms/src/console/Application.php(87): yii\console\Application->runAction('migrate/all', Array)
#44 /app/vendor/yiisoft/yii2/console/Application.php(147): craft\console\Application->runAction('migrate/all', Array)
#45 /app/vendor/yiisoft/yii2/base/Application.php(386): yii\console\Application->handleRequest(Object(craft\console\Request))
#46 /app/craft(22): yii\base\Application->run()
#47 {main}
Exception 'craft\errors\MigrateException' with message 'An error occurred while migrating Craft CMS.'

in /app/vendor/craftcms/cms/src/services/Updates.php:241

Stack trace:
#0 /app/vendor/craftcms/cms/src/console/controllers/MigrateController.php(259): craft\services\Updates->runMigrations(Array)
#1 [internal function]: craft\console\controllers\MigrateController->actionAll()
#2 /app/vendor/yiisoft/yii2/base/InlineAction.php(57): call_user_func_array(Array, Array)
#3 /app/vendor/yiisoft/yii2/base/Controller.php(157): yii\base\InlineAction->runWithParams(Array)
#4 /app/vendor/yiisoft/yii2/console/Controller.php(164): yii\base\Controller->runAction('all', Array)
#5 /app/vendor/yiisoft/yii2/base/Module.php(528): yii\console\Controller->runAction('all', Array)
#6 /app/vendor/yiisoft/yii2/console/Application.php(180): yii\base\Module->runAction('migrate/all', Array)
#7 /app/vendor/craftcms/cms/src/console/Application.php(87): yii\console\Application->runAction('migrate/all', Array)
#8 /app/vendor/yiisoft/yii2/console/Application.php(147): craft\console\Application->runAction('migrate/all', Array)
#9 /app/vendor/yiisoft/yii2/base/Application.php(386): yii\console\Application->handleRequest(Object(craft\console\Request))
#10 /app/craft(22): yii\base\Application->run()
#11 {main}

We have a client attempting to run this in an IIS environment. Does this plugin support those environments?

Not 100% sure the cause but it seems that if a job goes into the queue as "delayed", it seems to stall the queue from running and no jobs process.

For example, on order complete, as well as the job for sending an order email, we have a plugin, Klaviyo Connect, that pushes a job to the queue. However, this job goes in as delayed initially, then switches to pending, but the queue does not process any of the jobs.

If I then disable the Klaviyo Connect plugin, and make another order, the email job (and any others) process fine.

I don't believe it is the Klaviyo Connect plugin issue, but rather something linked to a delayed job - as when using the Craft queue, all jobs process correctly, including Klaviyo Connect.

Any thoughts?

I have installed this plugin, but I am not seeing normal queue jobs getting processed. Purchases in my store should send emails, and those jobs are sitting in the queue. If I run the "Async Queue Test", all of the jobs will then process. But I need these emails to get sent out immediately.

When I tested a 2nd order, I noticed the jobs from the first order got processed, but now my queue has jobs from this 2nd order sitting in the queue.

In my local environment, my PHP_BINARY is at /usr/bin/php, and in production, it is in /usr/local/bin/php. In both environments, my open_basedir is not set to anything.

How are the jobs being created during an order handled differently from the Async Queue Test jobs? Why do those get things flowing?

Thanks so much for your help!

Craft CMS: 3.7.24
Craft Async Queue: 2.3
Craft Commerce: 3.4.8

I've got a queue that's running and fetching data from a repository, running a bunch of python scripts to convert those files and finally running libreoffice's unoconv to output some pdfs.

When this plugin is running I can get halfway through a list of 250 items until the unoconv starts giving errors and stops producing the pdf files.

Without it the server becomes unresponsive until queue is complete and unoconv throws no errors and produces all the pdfs.

I'm thinking it may be a memory issue but not sure how to debug it.

Hello,

I recently upgraded from version 1.4 to 2.1.1, but I seem to be encountering a situation where jobs remain stuck at the Pending state.

I'm not seeing any error messages or anything pointing to a problem. Downgrading back to 1.4 resolves my issues.

Any help would be appreciated.

I think there is a typo in the documentation. I couldn't get this working. After some digging I found out there is no such env var named PATH_PHP_BINARY in the PhpExecutableFinder class: https://github.com/symfony/symfony/blob/master/src/Symfony/Component/Process/PhpExecutableFinder.php

Renamed it to PHP_PATH as it is stated there and everything worked fine.

I've installed AsyncQueue localy on MAMP and it is not runing.

But when I deactivate the plugin and then activate it again it runs smoothly. Why?

Craft CMS 3.1.28
AsyncQueue 2.0.0
PHP 7.2.14
Apache

No Errors in the queue.log

We're facing the problem that the parameter for the number of concurrent background processes – set in the environment file via ASYNC_QUEUE_CONCURRENCY=4 – is ignored at times. First, everything seems fine and the maximum number of reserved processes equals the number set in the env file. However, after a while (hard to say when exactly, but it happens) the number of concurrent processes starts to increase, bringing the CPU close to 100% and endangering the uptime of the website. This is particularly worrying when there are a lot of processes to be dealt with. Restarting PHP helps solve the issue, but isn't a feasible solution.

In the screenshots below, you can see that there's a limit of 4 set in the plugin, but there are 7 concurrent processes. We're running Craft 3.4.10.1 (latest as of today) with PHP 7.2 on a Web Server with Ubuntu 18.04.

PHP version | 7.4.20
Darwin 20.6.0
MySQL 5.7.34
Craft Pro 3.7.13

When I install via Plugin store I get:

Response:

PHP Fatal Error 'yii\base\ErrorException' with message 'Class GuzzleHttp\Psr7\Response contains 6 abstract methods and must therefore be declared abstract or implement the remaining methods (Psr\Http\Message\RequestInterface::getRequestTarget, Psr\Http\Message\RequestInterface::withRequestTarget, Psr\Http\Message\RequestInterface::getMethod, ...)'

I have a Commerce build that refused to send emails when this plugin was enabled. I'm using a custom template path (craft/src instead of craft/templates), and the emails would fail in the queue manager with an "HTML email template not found" error. As soon as I disabled the plugin, they all sent immediately without a hitch. We use Postmark to send SMTP mail. System emails sent just fine, but Commerce emails got stuck with this plugin enabled - one for two days, which is no bueno.

2022-04-14 15:44:00 [-][-][-][info][craft\queue\QueueLogBehavior::beforeExec]  [10608] Sending email for order #3657 (attempt: 1, pid: 305941) - Started
2022-04-14 15:44:00 [-][-][-][error][craft\commerce\services\Emails::sendEmail] Email template does not exist at “email/admin_order.html” which resulted in “email/admin_order.html” for email “Admin Order Notification”. Order: “f57631a”.
2022-04-14 15:44:00 [-][-][-][error][craft\queue\QueueLogBehavior::afterError]  [10608] Sending email for order #3657 (attempt: 1, pid: 305941) - Error (time: 0.062s): Email template does not exist at “email/admin_order.html” which resulted in “email/admin_order.html” for email “Admin Order Notification”. Order: “f57631a”.
2022-04-14 15:44:00 [-][-][-][error][craft\commerce\errors\EmailException] craft\commerce\errors\EmailException: Email template does not exist at “email/admin_order.html” which resulted in “email/admin_order.html” for email “Admin Order Notification”. Order: “f57631a”. in /srv/users/SITE_X/apps/SITE_X/craft/vendor/craftcms/commerce/src/queue/jobs/SendEmail.php:54
Stack trace:
#0 /srv/users/SITE_X/apps/SITE_X/craft/vendor/yiisoft/yii2-queue/src/Queue.php(246): craft\commerce\queue\jobs\SendEmail->execute(Object(craft\queue\Queue))
#1 /srv/users/SITE_X/apps/SITE_X/craft/vendor/yiisoft/yii2-queue/src/cli/Queue.php(162): yii\queue\Queue->handleMessage('10608', 'O:35:"craft\\com...', '300', '1')
#2 /srv/users/SITE_X/apps/SITE_X/craft/vendor/yiisoft/yii2-queue/src/cli/Command.php(146): yii\queue\cli\Queue->execute('10608', 'O:35:"craft\\com...', '300', '1', '305941')
#3 [internal function]: yii\queue\cli\Command->actionExec('10608', '300', '1', '305941')
#4 /srv/users/SITE_X/apps/SITE_X/craft/vendor/yiisoft/yii2/base/InlineAction.php(57): call_user_func_array(Array, Array)
#5 /srv/users/SITE_X/apps/SITE_X/craft/vendor/yiisoft/yii2/base/Controller.php(178): yii\base\InlineAction->runWithParams(Array)
#6 /srv/users/SITE_X/apps/SITE_X/craft/vendor/yiisoft/yii2/console/Controller.php(182): yii\base\Controller->runAction('exec', Array)
#7 /srv/users/SITE_X/apps/SITE_X/craft/vendor/yiisoft/yii2/base/Module.php(552): yii\console\Controller->runAction('exec', Array)
#8 /srv/users/SITE_X/apps/SITE_X/craft/vendor/yiisoft/yii2/console/Application.php(180): yii\base\Module->runAction('queue/exec', Array)
#9 /srv/users/SITE_X/apps/SITE_X/craft/vendor/craftcms/cms/src/console/Application.php(89): yii\console\Application->runAction('queue/exec', Array)
#10 /srv/users/SITE_X/apps/SITE_X/craft/vendor/yiisoft/yii2/console/Application.php(147): craft\console\Application->runAction('queue/exec', Array)
#11 /srv/users/SITE_X/apps/SITE_X/craft/vendor/yiisoft/yii2/base/Application.php(384): yii\console\Application->handleRequest(Object(craft\console\Request))
#12 /srv/users/SITE_X/apps/SITE_X/craft/craft(27): yii\base\Application->run()
#13 {main}

Hey @ostark,

I'm seeing frequent MySQL deadlocks when using this plugin to run Feed Me feeds in our dev environments.

If the Feed Me import config creates relationships to other elements, and I have ASYNC_QUEUE_CONCURRENCY set to anything > 1, I get a bunch of errors like so:

SQLSTATE[40001]: Serialization failure: 1213 Deadlock found when trying to get lock; try restarting transaction The SQL being executed was: INSERT INTO relations (fieldId, sourceId, sourceSiteId, targetId, sortOrder, dateCreated, dateUpdated, uid) VALUES (104, 246236, NULL, 31317, 1, '2019-11-11 14:47:58', '2019-11-11 14:47:58', '6ade060b-2f5c-4f6c-9f11-26d2f8afe40e'), (104, 246236, NULL, 19343, 2, '2019-11-11 14:47:58', '2019-11-11 14:47:58', '39c20ea7-b126-443d-93d9-e32d4dd6a489') - Schema.php: 664.

Craft CMS: 3.3.13 with the default (DB) queue driver
MySQL 5.7
craft-async-queue: 2.0.0

Happy to provide further details / reduced test case on request

Not currently seeing this in production so not a massive deal, but if you've any ideas I'm all ears :)

I get an error after 300 seconds (even though CLI timeout should be 0 and FPM timeout is set to 3600):

2021-03-10 13:57:51 [-][-][-][error][Symfony\Component\Process\Exception\ProcessTimedOutException] Symfony\Component\Process\Exception\ProcessTimedOutException: The process "'/usr/local/bin/php' 'craft' 'queue/exec' '241276' '300' '1' '0' '--color='" exceeded the timeout of 300 seconds. in /app/vendor/symfony/process/Process.php:1213

I'm a little unclear on how I should have this configured, @ostark -- can you advise?

Having installed the plugin and setup the correct PHP_BINARY I am getting an error with the Resaving Entries task after editing an entry type field layout. Happens both locally and on a staging site.

The error in the queue DB table states Calling unknown method: craft\console\Request::getIsPost().

When I re-run the task using the "Try again" option in the CP it works fine, which looks to go through the usual ajax task process.

Have you noticed anything like this already? Let me know if you need more info, logs etc.

Thanks!

I was trying to upgrade from 1.3.3 to the latest version 1.4 using php 7.2 and ran into the following:

$ php72 ~/bin/composer require ostark/craft-async-queue
    1/1:	http://repo.packagist.org/p/provider-latest$5f557240e6d1d5eac226b023415626a9a5a38ff74e238d68fb578696b121e8dc.json
    Finished: success: 1, skipped: 0, failure: 0, total: 1
Using version ^1.4 for ostark/craft-async-queue
./composer.json has been updated
Loading composer repositories with package information
Updating dependencies (including require-dev)
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Installation request for ostark/craft-async-queue ^1.4 -> satisfiable by ostark/craft-async-queue[1.4.0].
    - ostark/craft-async-queue 1.4.0 requires symfony/process ^4.0 -> satisfiable by symfony/process[4.0.x-dev, 4.1.x-dev, 4.2.x-dev, 4.3.x-dev, v4.0.0, v4.0.0-BETA1, v4.0.0-BETA2, v4.0.0-BETA3, v4.0.0-BETA4, v4.0.0-RC1, v4.0.0-RC2, v4.0.1, v4.0.10, v4.0.11, v4.0.12, v4.0.13, v4.0.14, v4.0.15, v4.0.2, v4.0.3, v4.0.4, v4.0.5, v4.0.6, v4.0.7, v4.0.8, v4.0.9, v4.1.0, v4.1.0-BETA1, v4.1.0-BETA2, v4.1.0-BETA3, v4.1.1, v4.1.2, v4.1.3, v4.1.4, v4.1.5, v4.1.6, v4.1.7, v4.1.8, v4.1.9, v4.2.0, v4.2.0-BETA1, v4.2.0-BETA2, v4.2.0-RC1, v4.2.1] but these conflict with your requirements or minimum-stability.

Digging deeper, it looks like symfony/process v4 has a requirement of php ^7.1

https://packagist.org/packages/symfony/process

I've tried installing symphony/process using php 7.2

$ php72 -v && php72 ~/bin/composer require symfony/process:^4.0
PHP 7.2.13 (cli) (built: Dec  7 2018 10:41:23) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies
    with Xdebug v2.6.0, Copyright (c) 2002-2018, by Derick Rethans
    with Zend OPcache v7.2.13, Copyright (c) 1999-2018, by Zend Technologies
    1/4: ........................................

  [InvalidArgumentException]
  Package symfony/process at version ^4.0 has a PHP requirement incompatible with your PHP version (7.0)

I just installed this plugin because I have a lot of images on my site so sometimes transforms cause timeouts, and when I login to my admin panel I get a really large number of transforms ending up in my queue, but the queue is not moving at all? This is on MAMP PHP 7.4.9

EDIT: duplicate of: craftcms/cms#1952

I'm using your plugin in combination with the Abandoned Cart for Craft Commerce 2 plugin, and I'm running into issues with the queue not piggybacking on HTTP requests when the AsyncQueue plugin is enabled. What I see in Utilities > Async Queue Test is that one job is waiting. I also see it as pending in the CP's left nav. It just hangs around as pending though and never completes even when clicking around in the admin to generate requests. As soon as I disabled AsyncQueue and create a new HTTP request, the jobs run. Because it works as expected when AsyncQueue is disabled, that leads me to think either something is wrong with AsyncQueue or I'm misunderstanding how it's supposed to work.

As I understand it, this plugin is not supposed to change how the jobs in the queue are triggered, it's only supposed to delegate them to a background process, right?

Craft CMS Version: 3.1.12
AsyncQueue Version: 2.0.0
Abandoned Cart for Craft Commerce 2 Version: 1.0.1
Host: fortrabbit

Tracked this (very weird) issue down to something with this plugin and some sort of change related to Symphony "Process"?

Since running a composer update going from Craft 3.7.19.1 to Craft 3.7.21, when editing any Entry there is now an error (screenshot attached) - until AsyncQueue is disabled; when things begin to work properly again.

Attached is a web.log of the output of this error when it happens. Andrew Welch managed to track it down as failing on this bit of code in the /cms/vendor/symphony/process/Process.php file

        $envPairs = [];
        foreach ($env as $k => $v) {
            if (false !== $v) {
                $envPairs[] = $k.'='.$v;
            }
        }

web.log.txt

Hi,
First of all thank you for this great plugin!

One small problem though, the composer/process package upgrade to 4.0 also bumped the minimum PHP version to 7.1.3. Was this intentional?

Ubuntu 16.04 has php 7.0.32 (which is the OS we're running on out servers) so this breaks our deployments.

Hey there! Any plans for updating for Craft 4?

Since upgrading various projects to Craft 3.4, the Queue Manager continually stacks up new pending tasks which never seem to run. I have had to disable Async Queue in order for that list to go.

Seems likely to be related to this breaking change: https://craftcms.com/guides/updating-plugins-for-craft-34#custom-queue-drivers

It would be nice to have an environment config (example: DISABLE_ASYNC_QUEUE) for cases where you have continuous integration (with Jenkins for example) and multiple servers (staging, preprod, prod), but some of the servers don't have proc_open enabled. With Schematic and now Craft 3.1 project config, the plugin will be installed automatically on all servers even if proc_open is not there.
Thanks!

So this isn't really an issue, and I'm sure @ostark you'll say "yeah, that's how it works" but I figured I'd note it for completeness.

The way the Craft job queue works is that it's a synchronous queue. I can push 100 jobs into the queue, and they'll be executed sequentially.

With the async queue plugin, if I push 100 jobs into the queue, each one spawns a new process, and they all execute in parallel.

The side-effect of that is that if you push a whole lot of jobs, or even a few jobs that are very compute-intensive, it can cause an excess of CPU usage/swap.

I realize that much of this is by design; but maybe it makes sense to allow the queue to function synchronously the way the Craft queue runs, but executed via CLI where the restrictions, timeouts, etc. are not a factor as they are and running it via php-fpm?

Here's a real-world scenario that I ran into:

My ImageOptimize plugin adds a job to the queue when a new image is uploaded, to create all of the image transforms necessary. If the user drags in 100 images to upload at once (as they are want to do), this will spawn hundreds of asynchronous queues running simultaneously, doing pretty intensive computations.

The server then is brought to its knees, proc_open() fork failed, etc. There's no limit on the number of these asynchronous queues that can be running in parallel.

I understand that the reason for this is that restarting the job in the Queue Manager is not making a new submission, as such the craft/queue run isn't executed until another job is submitted.

I don't expect there is a workaround for this, but wondered if it might be worth adding a button "Run queue" to the plugin?

Would just help user experience for non-developer site admins.

Unless you have a better idea? :)

The 'Deleting Stale Template Caches' job is failing and complaining about hitting a memory limit

The command "/opt/plesk/php/7.2/bin/php craft queue/exec "79272" "300" "1" "" --color=" failed. Exit Code: 255(Unknown error) Working directory: /var/www/vhosts/mydomain.com/httpdocs 

Output: 
================ Error Output: ================ 
PHP Fatal Error 'yii\base\ErrorException' with message 'Allowed memory size of 134217728 bytes exhausted (tried to allocate 12288 bytes)' 

By my calculations that's a 128M limit, but my php.ini file reads 512MB for memory_limit. I've since tried adding the Craft config 'phpMaxMemoryLimit' => '512M' but that doesn't seem to have made any difference.

I should add I've run the tests for Asyc Queue from Utilities and they run fine so I'm sure the right php binary is being located correctly.

Craft: 3.3.3
Craft-Async-Queue: 2.0.0

Any ideas? Thanks

Similar to your Relax plugin, I could not install v3.0.0 of your plugin on Craft v3.

Your requirements could not be resolved to an installable set of packages.
ostark/craft-async-queue 3.0.0 requires craftcms/cms ^4.0.0

I could however install it with v2.3.0, but of course would prefer to remain on latest version!

It seems that tasks are only ever run once by async queue. The cache of async-queue-pool seems to hit the max and never get less, making me assume that the decrement() function isn't being called.

This means when $this->getPool()->canIUse() is called, false is always return after the first tasks have been put in.

I have a module that generates a decent number of tasks so getting this working would be a massive help!

Thanks!

I want to install AsyncQueue on a fresh install under php 27.2.10

I get this error:

composer require ostark/craft-async-queue
Using version ^2.0 for ostark/craft-async-queue
./composer.json has been updated
Loading composer repositories with package information
Updating dependencies (including require-dev)
Your requirements could not be resolved to an installable set of packages.

Problem 1
- Installation request for ostark/craft-async-queue ^2.0 -> satisfiable by ostark/craft-async-queue[2.0.0].
- Conclusion: remove symfony/process v3.3.6
- ostark/craft-async-queue 2.0.0 requires symfony/process ^4.2.0 -> satisfiable by symfony/process[v4.2.0, v4.2.1, v4.2.2, v4.2.3].
- Can only install one of: symfony/process[v4.2.0, v3.3.6].
- Can only install one of: symfony/process[v4.2.1, v3.3.6].
- Can only install one of: symfony/process[v4.2.2, v3.3.6].
- Can only install one of: symfony/process[v4.2.3, v3.3.6].
- Installation request for symfony/process (installed at v3.3.6) -> satisfiable by symfony/process[v3.3.6].

I tried

composer config platform --unset

But I can't install it.

I just moved a site from one domain to another. Sadly, the urls of the images are all stored as full paths. Since the url changed, the plugin had to walk through all images, which took ages and was quite demanding for the server.

I think it would be a good solution to store the relative path after the siteUrl.

Got this message from composer on install.
Package container-interop/container-interop is abandoned, you should avoid using it. Use psr/container instead.

Anything I can do or are you working on a fix?

We've been seeing issues, especially when running FeedMe feeds, related to Mutex. For a while we thought it was related to craftcms/cms#10189 but have since stood up Redis and enabled it as the mutex driver for our app. We continued to see issues, and finally thought of disabling async queue. Now that we've done so, the errors are gone and tasks complete successfully.

Please let us know if there's anything additional we can provide to help!

Uncaught exception 'yii\base\Exception' with message 'Could not acquire a mutex lock for the queue (queue).' in /opt/app/vendor/craftcms/cms/src/queue/Queue.php:780

I installed the plugin on a Craft Pro 2.5.19.1 w/ Commerce 3.2.14

Checked the path that's in the Utilities piece - ensured it was correct by running which php

Clicked test, all the test jobs popped up, which is great. However, they just sit in a pending state and never actually run.

From here, I added the PHP_BINARY bit to my .env; composer remove-d, and then re-installed the plugin.

Repeated above steps, and still nada.

Any help would be appreciated - I'm sure I'm missing something silly here.

Tried installing this in a setup with Craft 3beta 30 on ubuntu 16.04 with php-fpm (php7.0.x). Installed it, tried triggering a queue job (by saving an article), but the jobs are stuck on "waiting" in the CP. They don't seem to be executed. My php binary is in /usr/bin. I'll gladly contribute more info on this issue, but I'm not really sure what you'd need.. Using ./craft queue/listen works – as do the AJAX/default handling.

Regards, Peter

I'm in the process of developing a plugin that will have a lot of jobs (emails, etc)

I was testing if this plugin helps with the jobs however they just get stuck in the 'pending' state.

The job runs fine when this plugin isn't enabled.

when i click the "run test" button, the test runs fine.

when I run nice /usr/local/bin/php craft queue/run > /dev/null 2>&1 & the queue starts to clear.

I've set PHP_BINARY to "/usr/local/bin/php"

no errors are showing up in the logs either.

runQueueAutomatically is true

Craft CMS: 3.7.37
Craft Async Queue: 2.3
Craft Commerce: n/a
PHP: 8.0

nb: this is running on nitro

AsyncQueue has stopped working on several sites after upgrading to Craft 4. When I run the AsyncQueue test, 10 jobs get added to the queue but none of them run after running ./craft queue/run in my terminal. When AsyncQueue is not installed, quick jobs do run after running the CLI command, including the AsyncQueue tests (after uninstalling and leaving the jobs there to be processed).

I do see this error when I run ./craft queue/run regardless of whether or not AsyncQueue is installed:

[ERROR] Queue must be an application component.
yii\base\InvalidConfigException

It's possible it's related, but I'm having a hard time tracking down the source of that error message.

this plugin does not work when the queue component is changed to redis in config/app.php file as follows:
'components' => [
'queue' => [
'class' => yii\queue\redis\Queue::class,
'redis' => 'redis', // Redis connection component or its config
'channel' => 'queue', // Queue channel key
],
],

Hi the command craft queue/run does not seem to be available in craft 3 RC15 ?
I downloaded it through the plugin store.

composer.json of the downloaded plugin:

{
  "name": "ostark/craft-async-queue",
  "description": "A queue handler that moves queue execution to a non-blocking background process",
  "type": "craft-plugin",
  "version": "1.3.0",
  "keywords": [
    "craft",
    "cms",
    "craftcms",
    "craft-plugin",
    "queue"
  ],
  "support": {
    "docs": "https://github.com/ostark/craft-async-queue/blob/master/README.md",
    "issues": "https://github.com/ostark/craft-async-queue/issues"
  },
  "license": "MIT",
  "authors": [
    {
      "name": "Oliver Stark",
      "homepage": "https://www.fortrabbit.com"
    }
  ],
  "require": {
    "craftcms/cms": "^3.0.0-RC1"
  },
  "autoload": {
    "psr-4": {
      "ostark\\AsyncQueue\\": "src/"
    }
  },
  "extra": {
    "name": "AsyncQueue",
    "handle": "async-queue",
    "hasCpSettings": false,
    "hasCpSection": false,
    "changelogUrl": "https://raw.githubusercontent.com/ostark/craft-async-queue/master/CHANGELOG.md"
  }
}

Hi,

I can’t figure out why Commerce emails aren’t being sent when this plugin is installed. I don’t even see them go through the Queue at all. If I disable the plugin, everything works as expected (when logged into the CP). I’ve simply installed the plugin without any config, am I missing something to get this to work?

Thanks!

AsyncQueue 2.1.1
Craft CMS: 3.5.12.1
Commerce: 3.2.7
PHP: 7.2.24

Hi,

I have just installed the plugin. For heavy task I constantly get this error:

The process "/RunCloud/Packages/php72rc/bin/php craft queue/exec "11530" "300" "1" "" --verbose=1 --color=" exceeded the timeout of 300 seconds.

How do I increase or disable the timeout? Thanks.

When we enable this plugin, our queue starts filling up with an infinitely growing list of image transforms. In each job:

Generating pending image transforms</h2>

ID: 352
Status: Pending
Progress: 0%
Time to reserve: 300 seconds
Priority: 2048
Pushed at: Jul 13, 2022, 7:30:42 AM

Job data:
{
    "description": null
}

The list just keeps growing and they never process/change from pending. Any thoughts?