aerendir / bundle-commands-queues Goto Github PK
View Code? Open in Web Editor NEWA bundle to manage queues and process jobs. Includes a never ending daemon.
License: MIT License
A bundle to manage queues and process jobs. Includes a never ending daemon.
License: MIT License
As suggested by best practices: Documentation.
As on
It is useful to make the developer able to adjust queues settings when the Daemon is still running.
This way it is possible to test various configurations without the need of a new deployment.
Once a good configuration is reached, it will be set in the config.yml
so on the next deploy it is available by default.
Be careful with trees of jobs.
Currently the config param is used in QueuesDaemon::processNextJob()
: if the max_concurrent_jobs
amount is reached, the method returns false
.
The method QueuesDaemon::processNextJob()
is used by RunCommand::execute()
(line 179) to check if the next job processing started or not.
If false
, the RunCommand
prints a message that says that queue is empty.
But this message is not always true, as the method QueuesDaemon::processNextJob()
may return false
also if other Jobs exist but in this moment they cannot be processed as the maximum number of them running concurrently is reached.
So, print from the QueuesDaemon::processNextJob()
method a more precise log for the console and completely remove any logging information from the RunCommand::execute()
method.
Also rationalize the way in which the initialization of new Jobs happen.
Maybe a method QueuesDaemon::canProcessAnotherJob()
is better: this method is used by RunCommand::execute()
foreach
at line 170 to decide if it has to pass to the next queue or continue with the current one.
More, it seems there is a bug: if max_concurrent_jobs
is set, for example, to 30
for at bundle level, the Daemon tries to initialize only 10
jobs, anyway.
This way will not be done a flush
operation each time an object is scheduled.
Save the PID in the database.
Then, when checking for stale jobs, check if the PID already exists and if not, then clean up the job.
A stale job is marked as INCOMPLETE
and if it can be retried, it will.
It has to be possible to define a Date after which the Job cn be executed. Before this date the Job has to be ignored by the Daemon.
Currently an array of Job
s is returned if any are found.
Make possible to decide which statuses the Jobs returned have to have.
Save the configuration of the daemon;
Do not link create a two way relation: only relate the jobs to the Worker, but not the worker to the Job.
Show the running workers in the interface.
The model class is currently ready to handle them, but the priorities has to be implemented.
config.yml
: Configuration is generally valid for each daemon
config.yml
: configuration valid for a single defined worker
command line: overwrite all the config.yml
defined configurations.
As on
Remove badges from README.md
Remove connection with CodeShip (on codeship and in GitHub)
It have to be possible to create a relation between jobs, so a Job MUST be executed after another Job completes successfully.
This is important to know the real environment used to run the Job
and, if the Job
is aware of its ID, also the passed ID.
The arguments are created in SerendipityHQ\Bundle\CommandsQueuesBundle\Service\JobsManager::createJobProcess()
.
Currently when a Job is created its options are sorted to make possible the check on their existence.
This is not good if the command to process uses arguments as their order is relevant.
The sorting algorithm MUST distinguish between arguments and options to correctly sort them.
Use three new methods:
Job::addArgument()
Job::addOption()
Job::addShortcut()
The purpose is to customize the ObjectManager
used by the bundle: it has to be different than the one used by the application, so it is possible to flush new jobs.
Currently this internal ObjectManager is created in @SHQCommandsQueues/Resources/config/orm.yaml
.
Here the @doctrine::getmanager()
method is used.
This method accepts the name of the ObjectManager
to use and if it is null
, then the default one is used.
So, as in @SHQCommandsQueues/DependencyInjection/Confguration
the default value for the config param is null
, it seems that the ObjectManager
returned is actually the default one, the one used by the rest of the application.
This is obviously wrong.
Do some tests to understand if a new ObjectManager
is used or if, in reality, the default one used by the application is (wrongly) used.
If detached, they caused an "Entity not found" error. Find a way to detach them safely, without triggering the error.
If a Job fails, it should be retried.
Is set to null
, the daemon will run indefinitely, until a SIGTERM
signal will be intercepted or a memory leak causes the crash.
The null
parameter has to be read by the configuration that sets it to 0.
Then Profiler::isMaxRuntimeReached()
will return false
is $this->maxRuntime
is 0
.
Anyway, it currently is not clear if it belongs to the bundle in general, or if it configures a daemon.
It is possible to configure it also for a single daemon.
but the real question is: is this config param really useful?
Maybe it is a better choice to completely remove it as there is no need to specify a max runtime.
Make possible to find Jobs based on the command and its arguments.
As suggested by Best practices: Tests.
As on
Currently before they are called it is required that the main while($daemon->isAlive())
cycle terminates and another one starts.
It is necessary that the check is done as soon as possible during the while
cycle.
Configure Travis-ci to test with both versions of Symfony and PHP.
Implement also other suggestions from best practices: Continuous integration.
As on
As it is possible to create more processes and fork them, then it has to be possible to create more than one worker directly from the bundle.
To each worker a configuration should be passed about which queues to process, idle times, max runtime, ecc.
So it will be possible to run multiple queue workers on one compute instance (see, for example, the worker instances on Heroku - the free plan offer only one free worker).
This opens to a whole new world of possibilities, where it is possible to set a queue worker to process heavy jobs and one to process light jobs. Also it will be possible to process in separate queue workers "live" jobs (required while the User is navigating the app) and "asynch" jobs that can be processed also in a second time.
SHQCommandsQueuesBundle
creates a default
worker that processes all existing queues;default
one;catch_all
is created anyway: this will process all the remaining queues that are eventually created.A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.