webador / slmqueuedoctrine Goto Github PK

Hi, I receive this error when I try to update my composer:

    - slm/queue-doctrine v0.4.1 requires doctrine/doctrine-orm-module ~0.7 -> no matching package found.
    - slm/queue-doctrine 0.4.0-beta3 requires doctrine/doctrine-orm-module ~0.7 -> no matching package found.
    - slm/queue-doctrine 0.4.0-beta2 requires doctrine/doctrine-orm-module ~0.7 -> no matching package found.
    - slm/queue-doctrine 0.4.0-beta1 requires doctrine/doctrine-orm-module ~0.7 -> no matching package found.
    - slm/queue-doctrine 0.4.0 requires doctrine/doctrine-orm-module ~0.7 -> no matching package found.
    - slm/queue-doctrine v0.4.1 requires doctrine/doctrine-orm-module ~0.7 -> no matching package found.
    - Installation request for slm/queue-doctrine 0.4.* -> satisfiable by slm/queue-doctrine[0.4.0, 0.4.0-beta1, 0.4.0-beta2, 0.4.0-beta3, v0.4.1].

This is my composer:

{
  "name": "Gamempire/gamempire",
  "description": "Skeleton of Gamempire",
  "license": "BSD-3-Clause",
  "keywords": [
    "framework",
    "zf2"
  ],
  "homepage": "http://framework.zend.com/",
  "minimum-stability": "dev",
  "require": {
    "php": ">=5.3.3",
    "firephp/firephp-core": "dev-master",
    "vlucas/phpdotenv": "1.*",
    "zendframework/zendframework": "2.4.*",
    "doctrine/doctrine-orm-module": "dev-master",
    "doctrine/orm": "dev-master",
    "gedmo/doctrine-extensions": "2.3.*",
    "zf-commons/zfc-twig": "1.*",
    "zf-commons/zfc-user-doctrine-orm": "0.1.*",
    "bjyoungblood/bjy-authorize": "1.4.*",
    "ezyang/htmlpurifier": "4.*",
    "tijsverkoyen/twitteroauth": "2.*",
    "facebook/php-sdk-v4": "4.0.*",
    "google/apiclient": "1.0.*@beta",
    "neilime/zf2-mobile-detect": "1.*",
    "exeu/apai-io": "1.*",
    "symfony/finder": "2.*",
    "slm/queue-doctrine": "0.4.*",
    "zendframework/zendservice-recaptcha": "2.*",
    "zendframework/zend-developer-tools": "1.*",
    "san/san-session-toolbar": "0.*",
    "asm89/twig-cache-extension": "~0.1",
    "rwoverdijk/assetmanager": "1.*"
  },
  "scripts": {
    "post-install-cmd": [
      "$(npm bin -q)/bower install"
    ],
    "post-update-cmd": [
      "$(npm bin -q)/bower install"
    ]
  },
  "autoload": {
    "psr-0": {
      "Gamempire": "module/Gamempire/src",
      "ApaiIO": "vendor/exeu/apai-io/lib"
    }
  }
}

What's wrong?

Thanks

Hi.
When can I expect a new release with updated composer dependencies from current master branch? I'm forced to use dev-master...

Ping @juriansluiman

I think we don't need the dependency to Doctrine ORM Module. Doctrine Module should be enough (we mostly rely on DBAL here).

Could somebody make the change ? @basz

EDIT : we should completely remove the dependency to DoctrineModule at all I think, and mimic the ZfcRbac configuration.

When i try to running default configuration on PHP 7.1 with this config:

'queue_manager' => [
'factories' => [
'default' => \SlmQueueDoctrine\Factory\DoctrineQueueFactory::class,
]
]

I got fatal error:

Fatal error: Declaration of SlmQueueDoctrine\Queue\DoctrineQueue::recover($executionTime): int must be compatible with SlmQueueDoctrine\Queue\DoctrineQueueInterface::recover(int $executionTime): int in /var/www/web/vendor/slm/queue-doctrine/src/Queue/DoctrineQueue.php on line 20

Good morning guys!
I'm having trouble running without a console.

php index.php queue doctrine <queueName> --start"

I am using "mezzio/mezzio-skeleton" + "JouwWeb/SlmQueueDoctrine" a configuration has been carried out successfully the queues are being raised. As you can see the console runs with MVC, but in the mezzio this is not working as expected.

Has anyone solved this problem using cli-blades? If so, which tube was used?

Sorry for my English, I used Google Translate

Hey,

I added two queues in my configuration. The default queue worker works fine. but if I start the worker for the email queue, I get the following result. But in my database table email_queue is only one job, which does not get executed. If I remove the MaxRunsStrategy the worker processes more and more jobs, which do not exist. What is wrong?

Console output by running php index.php queue doctrine default --start
Finished worker for queue 'default':

• 50 seconds passed
• 22.95MB memory usage
• 0 jobs processed
• lifetime of 50 seconds reached

Console output by running php index.php queue doctrine email --start
Finished worker for queue 'email':

• 30 jobs processed
• 0 seconds passed
• 22.75MB memory usage
• 30 jobs processed
• maximum of 30 jobs processed

Configuration

[
    'queues' => [
        'default' => [
            'table_name' => 'queue',
            'buried_lifetime' => 60 * 24 * 3,
            'deleted_lifetime' => 60
        ],

        'email' => [
            'table_name' => 'email_queue',
            'buried_lifetime' => 60 * 24 * 3,
            'deleted_lifetime' => 60
        ]
    ],

    'queue_manager' => [
        'factories' => [
            'default' => \SlmQueueDoctrine\Factory\DoctrineQueueFactory::class,
            'email' => \SlmQueueDoctrine\Factory\DoctrineQueueFactory::class,
        ]
    ],
        
    'worker_strategies' => [
        'queues' => [
            'default' => [
                \SlmQueue\Strategy\WorkerLifetimeStrategy::class => [
                    'lifetime' => 50
                ]
            ],
 
            'email' => [
                \SlmQueue\Strategy\WorkerLifetimeStrategy::class => [
                    'lifetime' => 50
                ],

                \SlmQueue\Strategy\MaxRunsStrategy::class => [
                    'max_runs' => 30
                ]
            ]
        ]
    ]
];

The Laminas Console package is deprecated, we should probably rethink the way we allow a user start this application. Could be done for example by the new https://github.com/laminas/laminas-cli library of Symonfy's counterpart.

Worker should catch PDOExceptions to avoid the worker from dying.

My 'supervised' workers are restarted three times in case of unexpected halting this however isn't enough to survive a simple mysql restart.

The application has thrown an exception!
PDOException
SQLSTATE[HY000] [2002] No such file or directory
.../vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOConnection.php:40

To the SlmQueueDoctrine users (ping @stefankleff, @basz) is anyone willing to write tests ? SlmQueueDoctrine is the only one not to have tests, and I'd like to have them so I can tag first version (0.2.0).

Thanks a lot !

I've been using some jobs that are self-enqueueing to run a job every so many minutes. I use a console command to seed the job into the queue and then it runs. As business requirements change, the delay between the job runs change. I made a console command parameter to pass in, updating the delay, but it results in two sets of the command in the queue running alongside each other. I was thinking it might make a nice feature to have a command to clear out any pending jobs given the job name. It would help out this specific case. (I'm not opposed to creating the PR, let me know if you think this fits appropriately in the project)

When adding job to queue with $queue->push($job);
There is an SQL exception :
SQLSTATE[23502]: Not null violation: 7 ERROR: null value in column "id" violates not-null constraint

I've read the following in the documentation:

"Warning : In previous versions of SlmQueueDoctrine the worker would quit if there where no jobs available for processing. That meant you could safely create a cronjob that would start a worker every minute. If you do that now you will quickly run out of available resources."

I'm curious as to why you would abandon this particular feature. Most/all of the queued jobs are pushed automatically, why then should someone manual run/process the queue every time? Could you perhaps add this feature back or at least explain why you abandoned it in the first place.

By the way if I'm missing something, maybe a new way to set up Cronjobs for SlimQueueDoctrine, could you share an example.
Thank you.

There is currently no way to hook into an event that can tell you whenever a job failed reliably. When a job crashes / exits / dies during executing a requests, it will remain on status busy.

Ideally there would be a system in place that can detect this, and that should broadcast an error event.

Maybe this should / could be fixed on the SlmQueue level.

Looping can cause infinite runs, see Webador/SlmQueue#35

Could someone explain how to run SlmQueueDoctrine worker with Zend Expressive?
Running php public/index.php queue default --start runs only main page of Zend Expressive application.

Is there a way to make the worker faster?

For example, I add 10 actions each second but the worker execute only 5 of them per second... in this way the queue_default table increase exponentially and it will be never empty.

How can I do?

Thanks

Sometimes it can happen, that script will allocate too much memory (eg. pdf generation) and then it throws uncatchable fatal error.

SlmQueueDoctrine recovers those jobs and runs it over and over, not firing up any event, and I can't find any way to handle those errors. I can set executionTime for recover, and they will not be recovered, but those jobs are still left unprocessed in database, and they just die silently.

What I need to do, is to not recover jobs, and set some things, eg. using Strategy. When memory is out, I can't do anything besides registering new register_shutdown_function, rising memory limit and do what I want. But this is reduntant, because I am already using Strategies to catch errors, send them to raygun or log them.

In this case nothing seems to work, script is stopped, slmqueue is trying to recover it, and it is stopped again. No event fired.

Is there any way to mark those jobs as finished with error, unrecoverable?

We already decided to transferhttps://github.com/juriansluiman/SlmQueue to the JouwWeb organization. We also actively use SlmQueueDoctrine ourselves. I think it makes sense to also transfer this one to us.

@basz and @juriansluiman Are you okay with this?

While executing a job, the job is removed even if the config mention
'queues' => array(
array('mails' => array(
'table_name' => 'queue_default',
'delete_lifetime' => 1440,
'buried_lifetime' => 5760,
'sleep_when_idle' => 2,
)
)
),

See Webador/SlmQueue#58. Commit would be the same like this one from SlmQueueBeanstalkd

See Webador/SlmQueueBeanstalkd#33

I want to schedule some work with a lower priority. However this is not possible with this adapter. And thus we resort to running multiple queue's: so that large chunks can run in a separate thread and will not block more important work.

The Beanstalk plugin does support this (https://github.com/juriansluiman/SlmQueueBeanstalkd#operations-on-queues).

Would a PR that introcues this be merged in this repo? Or is there a reason this is not implemented at this time?

In the README it is stated that:

This strategy will clear the ObjectManager before execution of individual jobs.

And that the ClearObjectManagerStrategy has a priority of 1000 (which would indeed make it run before individual jobs).

However, it actually runs after a job has been executed. This is clearly because the priority is actually set to -1000 (see the source).

Is there any plans to be compatible with SM3?

I could provide necessary PR.

I noticed entries like so '2016-06-21 21:26:41.000000' whilst they could be '2016-06-21 21:26:41.546421'

@basz:

Should update to 0.3.0-dev here:

https://github.com/juriansluiman/SlmQueueDoctrine/blob/master/src/SlmQueueDoctrine/Version.php

This should be bump to >=0.3.0-dev (like here: https://github.com/juriansluiman/SlmQueueSqs/pull/7/files#L3R30) :

https://github.com/juriansluiman/SlmQueueDoctrine/blob/master/composer.json#L27

Currently, it is impossible to use SlmQueue 0.7 with SlmQueueDoctrine, since in composer.json it is still fixed to version ^0.6.

However, they are compatible. Hence, it would be nice to upgrade the version constraint to 0.7.

Whenever I spin up multiple workers I get the following error every now and then:

Race-condition detected while updating item in queue

This effect is easily reproducible:

1. Spin up multiple workers
2. Remove the IdleNapStrategy (optional, but in this case it will directly crash)
3. Push some jobs on the queue

The problem is that for almost all database platforms no locking is used at all. That is due to the way we call DBAL:

$queryBuilder->from($platform->appendLockHint($this->options->getTableName(), LockMode::PESSIMISTIC_READ));

This will hint to use locking. However most platforms will not do anything with it. In my case I am using MariaDB which uses the MySql platform. And that does nothing with this hint:

START TRANSACTION;
SELECT * FROM queue_default WHERE (status = ?) AND (queue = ?) AND (scheduled <= ?) ORDER BY priority ASC, scheduled ASC LIMIT 1;
COMMIT;

We can fix this by using Doctrine's ORM functionallity to properly lock. I do not see an easy way to implement this without ORM; as DBAL does not contain functionality for it.

What are your thoughts on this @basz?

Looking at https://github.com/juriansluiman/SlmQueueDoctrine/blob/master/src/SlmQueueDoctrine/Queue/Table.php#L159 i wonder the following ...

Where scheduled is a datetime field, but the parameter is set to php's time() method which returns a unixtime stamp? This seem incorrect to me? To have this functioning we should hint the query method of the field type.

$stmt = $conn->executeQuery($select, array(static::STATUS_PENDING, $this->getName(), new Timestamp),
                                     array(Type::SMALLINT, Type::STRING, Type::DATETIME));

But besides this discrepancy, why should we want to NOT be able to pop jobs that have a scheduled datetime that is IN THE PAST? That does not seem correct behavior to me...

please advise

I think 'AND scheduled > ?' should be removed....

This may not be the best place to put. Nontheless I could not find a better location.

I am in the process of updating slmqueue to v4. The queue is populated with almost 20k records, so I wanted to preserve them. The solution as mentioned in the CHANGELOG does not work because it is non-trivial to run two versions of slmqueue next to each other.

Instead I chose to manually update the queue using a migration script. I ended up with: https://gist.github.com/roelvanduijnhoven/8eea449d255ca4b8cafb

As other people might face the same problem I though I would share it.

Hi,

Currently I'm migrating an Zend MVC application to Zend Expressive, which use this module.

SlmQueue is already Expressive "ready".
Anyone has used SlmQueueDoctrine with Zend Expressive? Or you are using another adapter (Sqs)?

Thanks.

IM

Great project

I am developing an online store in which trade game codes

When I receive multiple requests to the server to buy, these are executed simultaneously in PHP,

I want to avoid collisions and ship one by one.

With SmlQueue I can send one by one and not several at once?

If metadata is set upon job creation, that metadata is lost when the job is retrieved by a worker. I believe there are two origins of the problem, both in SlmQueueDoctrine\Queue\Queue, one in the call to pop(), and one in the call to delete(). Both of these call $this->createJob(...), but neither includes the original metadata from the meta, instead only including the job's ID.

i.e.

php index.php queue doctrine default --start &
php index.php queue doctrine default --start &
php index.php queue doctrine default --start &
php index.php queue doctrine default --start &
php index.php queue doctrine default --start &

Hey,

Good day.

Not sure what I am doing wrong. Getting a segmentation fault when call findBy or findAll functions. It seems doctrine borks but not sure why? Any idea how to debug further maybe?

config

    'slm_queue' => array(
        'queue_manager' => array(
            'factories' => array(
                'default' => 'SlmQueueDoctrine\Factory\DoctrineQueueFactory'
            )
        ),
        'queues' => array(
            'default' => array(

            )
        ),
        'job_manager' => array(
            'factories' => array(
                'IntellectualProperty\Job\GenerateExcelFileJob' => 'IntellectualProperty\Factory\Exporter\GenerateExcelFile',
                'IntellectualProperty\Job\SendEmailExcelFileJob' => 'IntellectualProperty\Factory\Exporter\SendEmailExcelFile'
            )
        )
    ),

factory

<?php
namespace IntellectualProperty\Factory\Exporter;

use IntellectualProperty\Job\GenerateExcelFileJob;
use Zend\ServiceManager\FactoryInterface;
use Zend\ServiceManager\ServiceLocatorInterface;

class GenerateExcelFile implements FactoryInterface
{
    /** @noinspection PhpPossiblePolymorphicInvocationInspection
     * @noinspection SpellCheckingInspection
     */
    public function createService(ServiceLocatorInterface $serviceLocator): GenerateExcelFileJob
    {
        $parentLocator = $serviceLocator->getServiceLocator();

        $instance = new GenerateExcelFileJob(
                $parentLocator->get('ApplicationLog'),
                $parentLocator->get('IntellectualPropertyExcelReport'),
                $parentLocator->get('config'),
                $parentLocator->get('FileUploadService')
                );
        $instance->setEm($parentLocator->get('doctrine.entitymanager.orm_default'));

        return $instance;
    }

}

job:

<?php
namespace IntellectualProperty\Job;

use Application\Entity\File\Document;
...
use IntellectualProperty\Utility\HardCoreDebugLogger;
use SlmQueue\Job\AbstractJob;
use SlmQueue\Queue\QueueAwareInterface;
use SlmQueue\Queue\QueueAwareTrait;
use Zend\Log\Logger;

/*
 * Generate excel file and then trigger another job to send email
 */
class GenerateExcelFileJob extends AbstractJob implements EntityManagerAwareInterface, QueueAwareInterface
{
    use EntityManagerAwareTrait;
    use QueueAwareTrait;

    protected $applicationLog = null;
    protected $excelReport = null;
    protected $config = null;
    private $fileManager;

    public function __construct(Logger $applicationLog, ExcelReport $excelReport, array $config = array(), FileManager $fileManager)
    {
        $this->applicationLog = $applicationLog;
        $this->config = $config;
        $this->excelReport = $excelReport;
        $this->fileManager = $fileManager;
    }

    // Execute
    public function execute(): bool
    {
          HardCoreDebugLogger::register('/tmp/trace.txt');
            $repository = $this->getEm()->getRepository('IntellectualProperty\Entity\Trademark');
            $rows = $repository->findAll();
            ...
    }
}

I have attached a trace file generated of what is accessed using this snippet https://gist.github.com/lyrixx/56dfc48fb7e807dd2a229813da89a0dc#file-hardcoredebuglogger-php. trace.txt

Regards.
Jarrett

PR to fix this, to follow.

Hey there!
I'm back at my old job and allowed/able to work on open source stuff again.
I just updated the packages to the newest version, but you've changed the serialization method in the current branch of slm/queue. Could you please update the code of this project as well?
Thank you!

Hi, I have created a factory called VisitJobFactory, a job called JobFactory and setted everything up in the configuration.

However, when i try to call the job, it says that I have redeclared the VisitJobFactory. Where I went wrong?

Here you can see the 4 files (job, factory, config and controller): https://gist.github.com/Gamempire/08444e1608f1e76047f6

Thanks a lot

PS: $jobManager = $serviceLocator->get('SlmQueue\Job\JobPluginManager');
PPS: I'm using SlmQueueDoctrine.

Hi, having jobs in the queue and starting the queue on the command line with public/index.php queue doctrine mail --start, send all the mails in the queue. But putting jobs some time later in the queue, I get the error: Could not read from host with a following error code in the trace https://gist.github.com/aburcheri/bf1522886b07bb3b37ec#file-read_from_host

SQL direct in the Table queue class like this should imho be removed and put in a separate abstraction class.

This will make the queue testable for unit tests (which I like) and the SQLite database is then required to validate the SQL generation of the "generator". This could even make it possible to have generators for Mongo etc as the connection resides in the queue and the adapter will create the specific database queries.

data and status are reserved word in mysql. Although I did not encounter problems while working with them, it might be safer to change these at some point.

I got this:

PDOException
SQLSTATE[22001]: String data, right truncated: 1406 Data too long for column 'message' at row 1

I think it should be a TEXT field rather.

https://github.com/juriansluiman/SlmQueueDoctrine/blob/master/composer.json#L28

Is pegged at
"slm/queue": "~0.4.0-beta3"

As https://github.com/juriansluiman/SlmQueue 0.4.0 has been released can this be updated.

The queue::pop method does a beginTransaction on the connection. If upon update it then detects a race condition the transaction is neither rolledback nor commited.

Exception thrown LogicException for a race-condition but catched is only DBALException.

Which makes me wonder anyways. How would I be able to rollback if DBAL is not working? Can we discuss exceptions related to DBAL and how that should be handled See #35

Is it possible to configure a WorkerClass (or a Worker Class Factory) to use with Doctrine Worker Controller, if the standard Worker should be adapted?

Or it should be done with new Controller (+Factory) and Worker (+Factory)?

DoctrineQueue::peek tries to create a job from data it does not have.

The fix is to just return null for when no row is found.

Once per week a cronjob creates puts approx. 2000 messages into the queue during one transaction in our system. We have 4 workers processing messages from this doctrine queue. Unfortunately, several workers usually end up in deadlocks when fetching these messags from the queue. I tried looking or solutions but did not manage to come up with anything, also did not find anything similar in the already existing issues, so I'll leave the exception information below.

Caught exception while processing queue; An exception occurred while executing 'SELECT * FROM queue_default WHERE (status = ?) AND (queue = ?) AND (scheduled <= ?) ORDER BY priority ASC, scheduled ASC LIMIT 1 FOR UPDATE' with params [1, "my_queue", "2020-07-13 04:00:25.690387"]:

SQLSTATE[40001]: Serialization failure: 1213 Deadlock found when trying to get lock try restarting transaction; An exception occurred while executing 'SELECT * FROM queue_default WHERE (status = ?) AND (queue = ?) AND (scheduled <= ?) ORDER BY priority ASC, scheduled ASC LIMIT 1 FOR UPDATE' with params [1, "my_queue", "2020-07-13 04:00:25.690387"]:

Hi,

https://github.com/juriansluiman/SlmQueueDoctrine/blob/master/composer.json#L27

I think you should enforce the dependency to 0.3.* of slm-queue while we don't reach 1.0. Between 0.2 and 0.3 we had already a lot of BC, so I suppose we still may not follow semantic versioning until 1.0.

Furthermore, I increased dependency of ZF to 2.2 in SlmQueue, should do the same here. Regarding DoctrineORMModule, I'd say bump the dependency too. 0.6 is super old.

Hi team,

Hope you’re fine.

I’m writing an application using ZF2, PostgreSQL, Doctrine.

For a newsletter sending process, I use SlmQueue.
I have 2 jobs :

• members synchronization with mailservice
• newsletter sending process

My problem is basic :
I want to stop the queue if the first job is not completely terminated !

To achieve that, I’m using the shared event manager such as the doc :

  Module.php 

  public function onBootstrap(MvcEvent $e)
    {    
        $sharedEventManager->attach('SlmQueue\Worker\WorkerInterface', WorkerEvent::EVENT_PROCESS_JOB, function (WorkerEvent $e)
        {
            // a problem has been detected on the running job
            $result = $e->getResult();
            if (WorkerEvent::JOB_STATUS_FAILURE === $result) {
                $job = $e->getJob();
                die(sprintf(
                    'Job #%s (%s) failed executing', $job->getId, get_class($job)
                ));
            }
         }, 1000);
    }

The jobs successfully failed but the program never come in the if() statement (getResult() = 0).
I verified the returned status of the processJob() method in DoctrineWorker.php file and it’s 3 (= JOB_STATUS_FAILURE)

Is this a bug ? Or maybe I’m totally wrong with my program ?

Thanks for your answer.

Regards,
Cedric