aerendir / bundle-commands-queues Goto Github PK

As suggested by best practices: Documentation.

As on

• Aerendir/bundle-aws-ses-monitor#53
• Aerendir/bundle-stripe#36

As on

• Aerendir/bundle-aws-ses-monitor#50
• Aerendir/bundle-stripe#33

It is useful to make the developer able to adjust queues settings when the Daemon is still running.

This way it is possible to test various configurations without the need of a new deployment.

Once a good configuration is reached, it will be set in the config.yml so on the next deploy it is available by default.

As on

• Aerendir/bundle-stripe#30
• Aerendir/bundle-aws-ses-monitor#47

https://github.com/symfony/recipes-contrib

As on

• Aerendir/bundle-stripe#20
• Aerendir/bundle-aws-ses-monitor#23

Be careful with trees of jobs.

Currently the config param is used in QueuesDaemon::processNextJob(): if the max_concurrent_jobs amount is reached, the method returns false.

The method QueuesDaemon::processNextJob() is used by RunCommand::execute() (line 179) to check if the next job processing started or not.

If false, the RunCommand prints a message that says that queue is empty.

But this message is not always true, as the method QueuesDaemon::processNextJob() may return false also if other Jobs exist but in this moment they cannot be processed as the maximum number of them running concurrently is reached.

So, print from the QueuesDaemon::processNextJob() method a more precise log for the console and completely remove any logging information from the RunCommand::execute() method.

Also rationalize the way in which the initialization of new Jobs happen.

Maybe a method QueuesDaemon::canProcessAnotherJob() is better: this method is used by RunCommand::execute() foreach at line 170 to decide if it has to pass to the next queue or continue with the current one.

More, it seems there is a bug: if max_concurrent_jobs is set, for example, to 30 for at bundle level, the Daemon tries to initialize only 10 jobs, anyway.

• Improve log messages
• Ensure the initialized jobs are as many as the desired

This way will not be done a flush operation each time an object is scheduled.

Save the PID in the database.

Then, when checking for stale jobs, check if the PID already exists and if not, then clean up the job.

A stale job is marked as INCOMPLETE and if it can be retried, it will.

It has to be possible to define a Date after which the Job cn be executed. Before this date the Job has to be ignored by the Daemon.

Currently an array of Jobs is returned if any are found.

Make possible to decide which statuses the Jobs returned have to have.

As on Aerendir/bundle-aws-ses-monitor#44

As on

• Aerendir/bundle-stripe#41
• Aerendir/bundle-aws-ses-monitor#59

Save the configuration of the daemon;
Do not link create a two way relation: only relate the jobs to the Worker, but not the worker to the Job.
Show the running workers in the interface.

• Save the running Daemon in the database;
• Show the running Daemons in the interface;
• Link the Jobs to the Daemon who is processing them;
• Check for the alive Daemons at regular intervals;
• Make the check interval time configurable.

The model class is currently ready to handle them, but the priorities has to be implemented.

config.yml: Configuration is generally valid for each daemon
config.yml: configuration valid for a single defined worker
command line: overwrite all the config.yml defined configurations.

As on

• Aerendir/bundle-stripe#39
• Aerendir/bundle-aws-ses-monitor#56

As on

• Aerendir/bundle-stripe#34

• Aerendir/bundle-aws-ses-monitor#51

• Remove badges from README.md

• Remove connection with CodeShip (on codeship and in GitHub)

It have to be possible to create a relation between jobs, so a Job MUST be executed after another Job completes successfully.

This is important to know the real environment used to run the Job and, if the Job is aware of its ID, also the passed ID.

The arguments are created in SerendipityHQ\Bundle\CommandsQueuesBundle\Service\JobsManager::createJobProcess().

As on

• Aerendir/bundle-aws-ses-monitor#55
• Aerendir/bundle-stripe#38

Currently when a Job is created its options are sorted to make possible the check on their existence.

This is not good if the command to process uses arguments as their order is relevant.

The sorting algorithm MUST distinguish between arguments and options to correctly sort them.

Use three new methods:

• Job::addArgument()
• Job::addOption()
• Job::addShortcut()

http://stackoverflow.com/questions/42459738/how-to-solve-a-memory-leak-with-parent-child-self-referencing-dependencies-in-do

The purpose is to customize the ObjectManager used by the bundle: it has to be different than the one used by the application, so it is possible to flush new jobs.

Currently this internal ObjectManager is created in @SHQCommandsQueues/Resources/config/orm.yaml.

Here the @doctrine::getmanager() method is used.

This method accepts the name of the ObjectManager to use and if it is null, then the default one is used.

So, as in @SHQCommandsQueues/DependencyInjection/Confguration the default value for the config param is null, it seems that the ObjectManager returned is actually the default one, the one used by the rest of the application.

This is obviously wrong.

Do some tests to understand if a new ObjectManager is used or if, in reality, the default one used by the application is (wrongly) used.

If detached, they caused an "Entity not found" error. Find a way to detach them safely, without triggering the error.

If a Job fails, it should be retried.

Is set to null, the daemon will run indefinitely, until a SIGTERM signal will be intercepted or a memory leak causes the crash.

The null parameter has to be read by the configuration that sets it to 0.

Then Profiler::isMaxRuntimeReached() will return false is $this->maxRuntime is 0.

Anyway, it currently is not clear if it belongs to the bundle in general, or if it configures a daemon.

It is possible to configure it also for a single daemon.

but the real question is: is this config param really useful?

Maybe it is a better choice to completely remove it as there is no need to specify a max runtime.

As on

• Aerendir/bundle-stripe#32
• Aerendir/bundle-aws-ses-monitor#49

Make possible to find Jobs based on the command and its arguments.

As on

• Aerendir/bundle-aws-ses-monitor#48
• Aerendir/bundle-stripe#31

As suggested by Best practices: Tests.

As on

• Aerendir/bundle-stripe#37
• Aerendir/bundle-aws-ses-monitor#54

Currently before they are called it is required that the main while($daemon->isAlive()) cycle terminates and another one starts.

It is necessary that the check is done as soon as possible during the while cycle.

• Automatically enable the integration
• List Jobs
• Show Job
• List daemons
• Show Daemon
• Jobs list: order by the newest to the oldest
• Filter jobs by status
• Other filtering options for Jobs
• Disable the mosaic view
• Create tabs with preset statuses

Configure Travis-ci to test with both versions of Symfony and PHP.

Implement also other suggestions from best practices: Continuous integration.

As on

• Aerendir/bundle-aws-ses-monitor#34
• Aerendir/bundle-stripe#24

As it is possible to create more processes and fork them, then it has to be possible to create more than one worker directly from the bundle.

To each worker a configuration should be passed about which queues to process, idle times, max runtime, ecc.

So it will be possible to run multiple queue workers on one compute instance (see, for example, the worker instances on Heroku - the free plan offer only one free worker).

This opens to a whole new world of possibilities, where it is possible to set a queue worker to process heavy jobs and one to process light jobs. Also it will be possible to process in separate queue workers "live" jobs (required while the User is navigating the app) and "asynch" jobs that can be processed also in a second time.

1. If no workers are defined, the SHQCommandsQueuesBundle creates a default worker that processes all existing queues;
2. If at least one worker exists, this will replace the default one;
3. If at least one worker exists and this defines at least one queue to process, then another worker catch_all is created anyway: this will process all the remaining queues that are eventually created.

https://github.com/sonata-project/SonataNotificationBundle