mmoreram / gearmanbundle Goto Github PK
View Code? Open in Web Editor NEWGearmanBundle for Symfony2
License: MIT License
GearmanBundle for Symfony2
License: MIT License
Current Scenario : Workers needs to fetch server parameter from config.yml by default instead of localhost:4730 as default server. If server parameter are not available in config.yml then it needs to be set by default to localhost:4730.
Benefit : If client and worker wants to connect to same server (my scenario and may be most of the time) defined by config.yml then there is no need to specify it's details again in worker annotations.
Otherwise, each time while updating job server, we need to take care of both configuration (host:port config setting should be not duplicated throughout application). If forgotten, will break system.
From below code snippet , it seems that default server has been set to localhost:4730 , not the one in config.yml.
/**
* Adds into worker all defined Servers.
* If any is defined, performs default method
*
* @param \GearmanWorker $gmworker Worker to perform configuration
* @param array $servers Servers array
*/
private function addServers(\GearmanWorker $gmworker, Array $servers)
{
if (!empty($servers)) {
foreach ($servers as $server) {
$gmworker->addServer($server['host'], $server['port']);
}
} else {
$gmworker->addServer();
}
}
Expected :
I hope, It make sense.
Kindly revert APAS.
Thanks,
I'm getting this error when I attempt to php app/console cache:clear --env=prod. I'm using Symfony 2.1 master and development branch of GearmanBundle.
[LogicException]
You must call one of in() or append() methods before iterating over a Finder.
Hi,
Provided documentation link is showing 404.
http://mmoreramerino.github.com/GearmanBundle/annotations.html
is it changed? Kindly, let me know.
Hi,
I have update my composer to your new branch.
After making following changes to configuration file,
gearman:
# Bundles will parsed searching workers
bundles:
# Name of bundle
SiteBundle:
# Bundle namespace
namespace: Stockopedia\SiteBundle
When I tried to update composer , getting following error,
Installing mmoreram/gearman-bundle (dev-master 4fcd8f1)
Cloning 4fcd8f1
Writing lock file
Generating autoload files
Updating the "app/config/parameters.yml" file.
[Symfony\Component\Config\Definition\Exception\InvalidConfigurationException]
Unrecognized options "namespace" under "gearman.bundles.SiteBundle"
Kindly suggest solution, if any.
This bundle should subscribe to tags
... to clear bundle configuration.
Enable profiler data-collector for Gearman client calls.
Could be nice to enable/disable this options through a parameter boolean. By default should be true.
Also must be documented.
The Gearman Protocol states that each packet header reserves 14 bytes automatically:
Also, that the entire packet header cannot exceed 128 bytes.
This means that the function name + unique id cannot exceed 114 bytes (114 + 14 = 128).
Assuming an md5 unique id is used for jobs (32 bytes), that leaves a maximum of 82 bytes for the function name. Function names greater than 82 bytes (again, assuming a 32 byte unique id) will cause the gearman hash table to become corrupted and will pass jobs to workers with invalid function names. This will cause the workers to fail. The actual error seen in GearmanBundle worker's is:
[ErrorException]
Warning: GearmanWorker::work(): gearman_worker_work(GEARMAN_INVALID_FUNCTION_NAME) Function not found -> libgearman/worker.cc:968 in src/vendor/mmoreram/gearman-bundle/Mmoreram/GearmanBundle/Service/GearmanExecute.php line 160
The only solution is to restart gearman and clear all jobs. When using a persistent storage, all persisted jobs need to be removed as well. To make matters worse, this crash only happens sporadically.
For example, if your function looks like:
Function name: XxxxxYyyyyyyAaaaaaZzzzzzAaaaaaBbbbbbbCcccccZzzzzzWorker~dddddEeeeeeeeFffffffGgggggg
(83 bytes)
unique id (md5) acbd18db4cc2f85cedef654fccc4a4d8
(32 bytes)
total = 129 bytes OVERFLOW!!! (115 bytes + 14 bytes}
... it will eventually cause failure.
Confirmed on gearmand .27
I propose that the GearmanBundle:
References:
I am using Symfony 2.1 and mmoreramerino/gearmanbundle (dev-development). I run into troubles when setting up my own workers in my own bundle:
I added my ExampleBundle to dev.yml and also the namespace \Acme\ExampleBundle
This bundle was generated using Symfony's console app, so a PHP unit test for the default controller was also generated. This lead to a fatal error when warming up Gearman's cache as PHPUnit was not available at this point of time :
php app/console gearman:cache:warmup
Class 'PHPUnit_Framework_TestCase' not found
class: GearmanCacheLoader: load method
Line: $reflClass = new \ReflectionClass($file['class']);
Curious where the wiki docs went for this project....
By default in GearmanBundle generate_unique_key
set true
. And when I execute next code my worker doesn't get all 500 jobs:
$gearman = $this->getContainer()->get('gearman');
for ($i = 0; $i < 500; $i++) {
$gearman->doHighBackgroundJob('TestTestBundleWorkersWorker~testB', 'value1');
}
When I set generate_unique_key: false
in config everything work fine.
Anyone with arguments against?
When setting the number of iterations for a job to 1, there will be 2 job executions.
This is the same for 2 iterations, which will result in 3 job executions.
I think it should based on the error I get when I send an exception back from the worker/job (below). Is that correct?
[Symfony\Component\Debug\Exception\ContextErrorException]
Catchable Fatal Error: Argument 2 passed to Symfony\Component\EventDispatcher\ContainerAwareEventDispatcher::dispatch() must be an instance of Symfony\Component\EventDispatcher\Event, instance of Mmoreram\GearmanBundle\Event\GearmanClientCallbackExceptionEvent given, called in ../vendor/mmoreram/gearman-bundle/Mmoreram/GearmanBundle/Service/GearmanCallbacks.php on line 143 and defined in ../vendor/symfony/symfony/src/Symfony/Component/EventDispatcher/ContainerAwareEventDispatcher.php line 163
Currently there is no way to "reuse" configuration. For example, in a test.yml
file, this does not work:
imports:
- { resource: dev.yml }
... and then one could override as-needed for the test
environment.
Ideally, configuration would be exposed as described in http://symfony.com/doc/master/cookbook/bundles/extension.html
This way we could use config.yml, config_dev.yml, config_test.yml, etc., with a gearman
namespace.
I have not identified the reason, but installation of this bundle via composer in Symfony 2.1.4-DEV fails due to an invalid library path.
the vendors folder path results in: /Mmoreramerino/GearmanBundle//Mmoreramerino/GearmanBundle/lib
This error doesn't seem to occur when installing other packages. Will provide more details as I have it.
Seems like the bundle uses serialize($params) before sending the data to the GearmanClient.
This is not very useful unless your worker are a php script but in a distributed environment, it may not.
My suggestion is to no serialize at all but rather throw an exception if the payload is not "Gearman friendly" and then let the user adding the jobs decide on serializing (I do prefer Json for example.)
If unique value is NULL, and unique_generation_if_null is true, generate unique value for every Client method.
One of last Doctrine2 changes make this bundle crash.
PHP Catchable fatal error: Argument 1 passed to Mmoreram\GearmanBundle\Service\GearmanCallbacks::__construct() must be an instance of Symfony\Component\EventDispatcher\EventDispatcher, instance of Symfony\Component\HttpKernel\Debug\TraceableEventDispatcher given, called in /var/www/projects/instantluxe.parsing/app/cache/dev/appDevDebugProjectContainer.php on line 1422 and defined in /var/www/projects/GearmanBundle/Service/GearmanCallbacks.php on line 46
Type Hinting should always be with an interface
Hi,
Can I move app/config/gearman/dev.yml into my customed bundle configuration directory
like MyBundle/Resources/config/gearman.yml, or move it into sf2 configuration file like app/config_dev.yml, app/config_prod.yml?
Would GearmanBundle support this function?
Thanks.
All new developments and issues solves will be committed in a development branch. If developers do tests with these commits and assures a little their correctness, will be merged to stable branches
In documentation testworker class is created in GearmanBundle itself which will get overwritten if we update composer and will loose any business login inside bundle's worker class job.
I want to create worker class in my application bundle for writing jobs which can be used by gearman service same way it is using test worker class.
I am stuck , can anyone help me with that ?
Convert all available Task Callbacks to valid Symfony2 kernel events
Create a set of tools for easy Gearman metrics. An easy way to check any defined Queue values, jobs and some other stats.
Gearman users, what do you think about this idea?
Please, tell me your needs!
Hi,
I'd like to get a feedback from a worker during a long job.
In client I have (in a command with service container)
$dispatcher = $this->getContainer()->get('event_dispatcher');
$dispatcher->addListener("gearman.client.status", array($this, 'onStatus'));
$this->getContainer()->get('gearman')->doJob(...);
in worker
$job->sendData("data: ciao");
The event seems never triggered. Is this approach correct?
Thanks, regards
When I execute the following code:
$Gearman = $this->get('gearman');
$Gearman->doNormalJob('BundleName~test');
My code throws the following exception:
GearmanClient::doNormal() expects parameter 2 to be string, array given
Stack:
#0 [internal function]: Symfony\Component\Debug\ErrorHandler->handle(2, 'GearmanClient::...', '/Users/reneters...', 178, Array)
#1 /Users/reneterstegen/Sites/core.xxx.nl/vendor/mmoreram/gearman-bundle/Mmoreram/GearmanBundle/Service/GearmanClient.php(178): GearmanClient->doNormal('XXXBundleCoreBu...', Array, NULL)
#2 /Users/reneterstegen/Sites/core.xxx.nl/vendor/mmoreram/gearman-bundle/Mmoreram/GearmanBundle/Service/GearmanClient.php(153): Mmoreram\GearmanBundle\Service\GearmanClient->doEnqueue(Array, Array, 'doNormal', NULL)
#3 /Users/reneterstegen/Sites/core.xxx.nl/vendor/mmoreram/gearman-bundle/Mmoreram/GearmanBundle/Service/GearmanClient.php(266): Mmoreram\GearmanBundle\Service\GearmanClient->enqueue('XXXBundleCoreBu...', Array, 'doNormal', NULL)
#4 /Users/reneterstegen/Sites/core.xxx.nl/src/XXX/Bundle/CoreBundle/Controller/TestController.php(32): Mmoreram\GearmanBundle\Service\GearmanClient->doNormalJob('XXXBundleCoreBu...')
#5 [internal function]: XXX\Bundle\CoreBundle\Controller\TestController->testAction()
#6 /Users/reneterstegen/Sites/core.xxx.nl/app/bootstrap.php.cache(2815): call_user_func_array(Array, Array)
#7 /Users/reneterstegen/Sites/core.xxx.nl/app/bootstrap.php.cache(2789): Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object(Symfony\Component\HttpFoundation\Request), 1)
#8 /Users/reneterstegen/Sites/core.xxx.nl/app/bootstrap.php.cache(2918): Symfony\Component\HttpKernel\HttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true)
#9 /Users/reneterstegen/Sites/core.xxx.nl/app/bootstrap.php.cache(2220): Symfony\Component\HttpKernel\DependencyInjection\ContainerAwareHttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true)
#10 /Users/reneterstegen/Sites/core.xxx.nl/web/app_dev.php(19): Symfony\Component\HttpKernel\Kernel->handle(Object(Symfony\Component\HttpFoundation\Request))
#11 {main}
When I go down to the code the exception is thrown in this function:
/**
* Execute a GearmanClient call given a worker, params and a method.
*
* If he GarmanClient call is asyncronous, result value will be a handler.
* Otherwise, will return job result.
*
* @param array $worker Worker definition
* @param mixed $params Parameters to send to job
* @param string $method Method to execute
* @param string $unique A unique ID used to identify a particular task
*
* @return mixed Return result of the GearmanClient call
*/
private function doEnqueue(Array $worker, $params = '', $method = null, $unique = null)
{
$gearmanClient = new \GearmanClient();
$this->assignServers($gearmanClient);
return $gearmanClient->$method($worker['job']['realCallableName'], $params, $unique);
}
Caused by:
public function doNormalJob($name, $params = array(), $unique = null)
{
return $this->enqueue($name, $params, GearmanMethods::GEARMAN_METHOD_DONORMAL, $unique);
}
Here the default value for $params = array()
. In the rest of the chain this parameter is not altered so an array will be given to de doNormal method.
This looks like a bug.
Thanks in advance!
I am getting this error when using the command:
[Symfony\Component\Debug\Exception\ContextErrorException]
Notice: Undefined index: host in vendor/mmoreram/gearman-bundle/Mmoreram/GearmanBundle/Service/GearmanDescriber.php line 159
The data in $worker[servers] is:
array(1) {
'localhost' =>
array(2) {
'hostname' =>
string(9) "127.0.0.1"
'port' =>
string(4) "4730"
}
}
The configuration has a default value for "hostname" and not "host".
Hi,
We are facing some problem with your updated bundle logic.
Scenario -
We want to process on-demand thousand jobs. We want to create workers on-demand which will be alive once started on it own ( not with help of any Supervisord). Then we will terminate them after completing all jobs. ( We will manually run cron to kill them.)
In you previous bundle when we used to set iteration = 0, worker used to be stay alive till get killed manually.
But now for iteration = 0, after processing one job it is shutting down.
Things not possible,
As we don't know exact no of jobs , cannot hard code value in iteration annotation to let worker/job know exact it's iteration.
If we set iteration = 0 and put it under Supervisord then after completing one job worker will shutdown and Supervisord will try to restart them. So worker can't be killed.
But we don't want multiple workers running in background after all completing their jobs.We will manually run cron to kill them.
Kind integrate your previous logic which was good, flexible and was not keeping any Supervisord kind of dependency.
$alive = false;
if(iteration == 0){
$alive = true;
}
while ($gearmanWorker->work()) {
if ($gearmanWorker->returnCode() != GEARMAN_SUCCESS) {
break;
}
if($alive === false){
if ($iterations-- <= 0) {
break;
}
}
}
Kind revert ASAP ..Thank you.
Hi mmoreram,
For below setting,
type: file_system
After running any command second time getting error,
PHP Fatal error: Call to undefined method Doctrine\Common\Cache\FilesystemCache::get() in /var/www/prototype/Symfony/vendor/mmoreram/gearman-bundle/Mmoreram/GearmanBundle/Service/GearmanCacheWrapper.php on line 150
but for ,
type: array
working fine.
any idea ?
Getting this when calling a job with Gearmand 0.34
Deprecated: GearmanClient::do() [gearmanclient.do]: Use GearmanClient::doNormal() in /vhosts/domains/cardsinthepost.com/public/current/vendor/bundles/Mmoreramerino/GearmanBundle/lib/Mmoreramerino/GearmanBundle/Service/GearmanClient.php line 138
I think client callback are missing..
$gmc->setCreatedCallback("reverse_created");
$gmc->setDataCallback("reverse_data");
$gmc->setStatusCallback("reverse_status");
$gmc->setCompleteCallback("reverse_complete");
$gmc->setFailCallback("reverse_fail");
if not then how can set them ?
I'm trying out the following annotation
@gearman\Work(name = "TariochEveapiFetcherEveWorker", service = "tarioch.eveapi.worker.eve")
but an worker:list returns
@Worker: Tarioch\EveapiFetcherBundle\Component\Worker\EveWorker
callablename: TariochEveapiFetcherBundleComponentWorkerTariochEveapiFetcherEveWorker
Shouldn't my name be taken instead of the auto generated one from the path?
php app/console gearman:job:execute MyBundleWorkersemailWorker~sendMail --no-interaction
Results in dev
being outputted when performing this command from within the job:
var_dump($this->container->get('kernel')->getEnvironment());
Same thing for:
php app/console gearman:job:execute MyBundleWorkersemailWorker~sendMail --no-interaction --env="dev"
However, this fails:
php app/console gearman:job:execute MyBundleWorkersemailWorker~sendMail --no-interaction --env="prod"
With this error:
[Mmoreramerino\GearmanBundle\Exceptions\JobDoesNotExistException] GearmanBundle can't find job with name MyBundleWorkersemailWorker~sendMail
Kindly correct spelling mistake,
ADDTASKNACKGROUND => ADDTASKBACKGROUND
Gearmanclient.php
public function addTaskBackground($name, $params =array(), &$context = null, $unique = null)
{
$this->enqueueTask($name, $params, $context, $unique, GearmanMethods::GEARMAN_METHOD_ADDTASKNACKGROUND);
return $this;
}
The way of reducing this issue is
Generators\UniqueJobIdentifierGenerator
instead of
Generator\UniqueJobIdentifierGenerator
(no s with Generator)
Hello,
Is there a way to get the queue length?
Indeed, I run the method doBackgroundJob when I want to execute an action but I got a problem if the worker isn't launched : my queue increases without any reason. So I'd like to know the queue length in order to avoid useless doBackgroundJob.
Thanks for the answer
Right now one place I am using this has multiple environments on the same server, ie dev and staging. I'm not sure the best way to differentiate workers to pull only live jobs vs dev jobs. I'm trying to avoid running multiple instances of gearmand at the moment.
What if you could specify some sort of prefix which all workers/clients could automatically add to their calls? This would let you basically prefix all staging jobs with staging_Foo and dev with dev_Foo etc. Am not sure if this is the best way, and am open to suggestions for how to do it better.
Thanks, and awesome bundle.
Hi mmoreram,
It is good to have your bundle for gearman.
But i am little disappointed that, you closed my open issue #40 before getting confirmation of solved from my side.
Once again i reversed engineered your code and found that,
As per your Documentation configure should have
namespace: Mmoreramerino\TestBundle
But for your code to work it should be,
name: MmoreramerinoTestBundle
Also make sure to solved open issue before closing them. So that developer will get proper solution.
Whenever I get the gearman service it gave me a:
[ReflectionException]
Class does not exist
I got this because of the GearmanCacheWrapper which finds all the files in my bundle, and then checks for the class namespaces, even of non-php files.
lines 268 and 269 of GearmanCacheWrapper.php:
$classNamespace = $this->getFileClassNamespace($file->getRealpath());
$reflClass = new ReflectionClass($classNamespace);
$classNamespace was empty for alot of files and it created this exception.
Quick fix for me was to add:
if( !$classNamespace )
continue;
before the creation of the ReflectionClass.
Any idea how to fix this cleaner as I do not want to edit the vendor file?
This method should be able ( overridable )
Hi,
Below is my gearman client related code,
I wanna ask what's the format of the variable $job (the first argument of doJob function),
thanks.
$gmclient = $this->getContainer()->get('gearman');
$gmclient->setServer("172.16.6.1");
$record = array("msg" => "ok");
$gmclient->doJob($job, $record);
Could anyone please test connection to different gearman server rather than default?
For example i have config:
defaults:
servers:
cache:
hostname: 192.168.0.223
port: 4730
But client always tries to connect 127.0.0.1
$gearman = $this->getContainer()->get('gearman');
I know the problem is not in the config itself, because when i dump:
$gearman->getSettings()
I got all the server config correctly, so right now my temporary fix i could think of to get things work is:
$gearman = $this->getContainer()->get('gearman');
$settings = $gearman->getSettings()['defaults']['servers']['cache'];
$gearman->setServer($settings['hostname']);
$result = $gearman->doBackgroundJob....
Thank you.
When defining servers in app/config/config.yml
everything works as expected. However, if different settings exist in app/config/config_prod.yml
the workers fail to communicate correctly with gearmand. In this case the workers display suspicious configuration output:
@worker\servers :
#task1 - 10.57.23.24:4730
#0 - 127.0.0.1:4730
@Worker\description :
Works sms-related jobs
The line above - #0 - 127.0.0.1:4730
should not exist. What is it? The workers attempt to connect to local host but cannot because gearmand is on a separate machine, which results in this error:
[GearmanException]
Failed to set exception option
The only temporary solution is to remove gearman settings from all files except for config.yml.
PHP Catchable fatal error: Argument 1 passed to Mmoreram\GearmanBundle\Service\GearmanCacheWrapper::__construct() must be an instance of Mmoreram\GearmanBundle\Service\GearmanParser, instance of AppKernel given, called in /var/www/hpl/evetool/app/cache/prod/appProdProjectContainer.php on line 741 and defined in /var/www/hpl/evetool/vendor/mmoreram/gearman-bundle/Mmoreram/GearmanBundle/Service/GearmanCacheWrapper.php on line 78
PHP Stack trace:
PHP 1. {main}() /var/www/hpl/evetool/app/console:0
PHP 2. Symfony\Component\Console\Application->run() /var/www/hpl/evetool/app/console:22
PHP 3. Symfony\Bundle\FrameworkBundle\Console\Application->doRun() /var/www/hpl/evetool/vendor/symfony/symfony/src/Symfony/Component/Console/Application.php:121
PHP 4. Symfony\Component\Console\Application->doRun() /var/www/hpl/evetool/vendor/symfony/symfony/src/Symfony/Bundle/FrameworkBundle/Console/Application.php:96
PHP 5. Symfony\Component\Console\Application->doRunCommand() /var/www/hpl/evetool/vendor/symfony/symfony/src/Symfony/Component/Console/Application.php:191
PHP 6. Symfony\Component\Console\Command\Command->run() /var/www/hpl/evetool/vendor/symfony/symfony/src/Symfony/Component/Console/Application.php:904
PHP 7. Symfony\Bundle\FrameworkBundle\Command\CacheWarmupCommand->execute() /var/www/hpl/evetool/vendor/symfony/symfony/src/Symfony/Component/Console/Command/Command.php:244
PHP 8. Symfony\Component\DependencyInjection\Container->get() /var/www/hpl/evetool/vendor/symfony/symfony/src/Symfony/Bundle/FrameworkBundle/Command/CacheWarmupCommand.php:58
PHP 9. appProdProjectContainer->getCacheWarmerService() /var/www/hpl/evetool/app/bootstrap.php.cache:1973
PHP 10. Symfony\Component\DependencyInjection\Container->get() /var/www/hpl/evetool/app/cache/prod/appProdProjectContainer.php:420
PHP 11. appProdProjectContainer->getGearman_Cache_WrapperService() /var/www/hpl/evetool/app/bootstrap.php.cache:1973
PHP 12. Mmoreram\GearmanBundle\Service\GearmanCacheWrapper->__construct() /var/www/hpl/evetool/app/cache/prod/appProdProjectContainer.php:741
The following command returns an exception when configuration is different than the test worker. I'm assuming it is a problem with the autoloading.
php app/console gearman:worker:list
Results in:
Fatal error: Class 'PHPUnit_Framework_TestCase' not found in /var/www/project/src/vendor/symfony/src/Symfony/Bundle/FrameworkBundle/Test/WebTestCase.php on line 24
For example, in app/config/gearman/dev.yml
the following outputs worker information:
bundles:
GearmanBundle:
namespace: Mmoreramerino\GearmanBundle
active: true
However, putting the same testWorker.php
in a different bundle under a Workers
folder, the gearman:worker:list
command produces the exception. For example:
bundles:
GearmanBundle:
namespace: Mmoreramerino\GearmanBundle
active: false
ProjectBundle:
namespace: MyProject\Bundle\MyBundle
active: true
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.