Curious where the wiki docs went for this project....

Hi,
I have update my composer to your new branch.
After making following changes to configuration file,

gearman:
    # Bundles will parsed searching workers
    bundles:
    # Name of bundle
    SiteBundle:

        # Bundle namespace
        namespace: Stockopedia\SiteBundle

When I tried to update composer , getting following error,

Installing mmoreram/gearman-bundle (dev-master 4fcd8f1)
Cloning 4fcd8f1

Writing lock file
Generating autoload files
Updating the "app/config/parameters.yml" file.

[Symfony\Component\Config\Definition\Exception\InvalidConfigurationException]
Unrecognized options "namespace" under "gearman.bundles.SiteBundle"

Kindly suggest solution, if any.

If unique value is NULL, and unique_generation_if_null is true, generate unique value for every Client method.

All new developments and issues solves will be committed in a development branch. If developers do tests with these commits and assures a little their correctness, will be merged to stable branches

When defining servers in app/config/config.yml everything works as expected. However, if different settings exist in app/config/config_prod.yml the workers fail to communicate correctly with gearmand. In this case the workers display suspicious configuration output:

    @worker\servers :

        #task1 - 10.57.23.24:4730
        #0 - 127.0.0.1:4730

    @Worker\description :

        Works sms-related jobs

The line above - #0 - 127.0.0.1:4730 should not exist. What is it? The workers attempt to connect to local host but cannot because gearmand is on a separate machine, which results in this error:

  [GearmanException]              
  Failed to set exception option

The only temporary solution is to remove gearman settings from all files except for config.yml.

I am using Symfony 2.1 and mmoreramerino/gearmanbundle (dev-development). I run into troubles when setting up my own workers in my own bundle:

I added my ExampleBundle to dev.yml and also the namespace \Acme\ExampleBundle

This bundle was generated using Symfony's console app, so a PHP unit test for the default controller was also generated. This lead to a fatal error when warming up Gearman's cache as PHPUnit was not available at this point of time :

php app/console gearman:cache:warmup
Class 'PHPUnit_Framework_TestCase' not found

class: GearmanCacheLoader: load method
Line: $reflClass = new \ReflectionClass($file['class']);

Seems like the bundle uses serialize($params) before sending the data to the GearmanClient.

This is not very useful unless your worker are a php script but in a distributed environment, it may not.

My suggestion is to no serialize at all but rather throw an exception if the payload is not "Gearman friendly" and then let the user adding the jobs decide on serializing (I do prefer Json for example.)

The Gearman Protocol states that each packet header reserves 14 bytes automatically:

• 4 byte MAGIC CODE
• 1 byte null
• 4 byte TYPE
• 1 byte null
• 4 byte SIZE

Also, that the entire packet header cannot exceed 128 bytes.

This means that the function name + unique id cannot exceed 114 bytes (114 + 14 = 128).

Assuming an md5 unique id is used for jobs (32 bytes), that leaves a maximum of 82 bytes for the function name. Function names greater than 82 bytes (again, assuming a 32 byte unique id) will cause the gearman hash table to become corrupted and will pass jobs to workers with invalid function names. This will cause the workers to fail. The actual error seen in GearmanBundle worker's is:

  [ErrorException]                                                                                                                                                                                  
  Warning: GearmanWorker::work(): gearman_worker_work(GEARMAN_INVALID_FUNCTION_NAME) Function not found -> libgearman/worker.cc:968 in src/vendor/mmoreram/gearman-bundle/Mmoreram/GearmanBundle/Service/GearmanExecute.php line 160    

The only solution is to restart gearman and clear all jobs. When using a persistent storage, all persisted jobs need to be removed as well. To make matters worse, this crash only happens sporadically.

For example, if your function looks like:

Function name: XxxxxYyyyyyyAaaaaaZzzzzzAaaaaaBbbbbbbCcccccZzzzzzWorker~dddddEeeeeeeeFffffffGgggggg (83 bytes)
unique id (md5) acbd18db4cc2f85cedef654fccc4a4d8 (32 bytes)
total = 129 bytes OVERFLOW!!! (115 bytes + 14 bytes}

... it will eventually cause failure.

Confirmed on gearmand .27

I propose that the GearmanBundle:

1. Provide a way to specify a custom name for a function instead of forcing the user to use the auto-generated names.
2. Throw an error when a job/task is dispatched to gearman that exceeds 114 bytes for function name + unique id.

References:

• http://gearman.org/protocol/
• https://bugs.launchpad.net/gearmand/+bug/833394
• https://bugs.php.net/bug.php?id=59195

see https://scrutinizer-ci.com/g/mmoreram/GearmanBundle/inspections/0f2a2e9e-9950-4b9e-8cfd-cd007fd7ebf0/issues/files/Service/GearmanCacheWrapper.php?status=existing#inspectioncomment-21767185

I am getting this error when using the command:

  [Symfony\Component\Debug\Exception\ContextErrorException]                                                                                                       
  Notice: Undefined index: host in vendor/mmoreram/gearman-bundle/Mmoreram/GearmanBundle/Service/GearmanDescriber.php line 159

The data in $worker[servers] is:

array(1) {
  'localhost' =>
  array(2) {
    'hostname' =>
    string(9) "127.0.0.1"
    'port' =>
    string(4) "4730"
  }
}

The configuration has a default value for "hostname" and not "host".

I think it should based on the error I get when I send an exception back from the worker/job (below). Is that correct?

[Symfony\Component\Debug\Exception\ContextErrorException]                                                                                                                                                                                  
  Catchable Fatal Error: Argument 2 passed to Symfony\Component\EventDispatcher\ContainerAwareEventDispatcher::dispatch() must be an instance of Symfony\Component\EventDispatcher\Event, instance of Mmoreram\GearmanBundle\Event\GearmanClientCallbackExceptionEvent given, called in ../vendor/mmoreram/gearman-bundle/Mmoreram/GearmanBundle/Service/GearmanCallbacks.php on line 143 and defined in ../vendor/symfony/symfony/src/Symfony/Component/EventDispatcher/ContainerAwareEventDispatcher.php line 163 

In documentation testworker class is created in GearmanBundle itself which will get overwritten if we update composer and will loose any business login inside bundle's worker class job.

I want to create worker class in my application bundle for writing jobs which can be used by gearman service same way it is using test worker class.

I am stuck , can anyone help me with that ?

php app/console gearman:job:execute MyBundleWorkersemailWorker~sendMail --no-interaction

Results in dev being outputted when performing this command from within the job:

var_dump($this->container->get('kernel')->getEnvironment());

Same thing for:

php app/console gearman:job:execute MyBundleWorkersemailWorker~sendMail --no-interaction --env="dev"

However, this fails:

php app/console gearman:job:execute MyBundleWorkersemailWorker~sendMail --no-interaction --env="prod"

With this error:

[Mmoreramerino\GearmanBundle\Exceptions\JobDoesNotExistException] GearmanBundle can't find job with name MyBundleWorkersemailWorker~sendMail

I have not identified the reason, but installation of this bundle via composer in Symfony 2.1.4-DEV fails due to an invalid library path.

the vendors folder path results in: /Mmoreramerino/GearmanBundle//Mmoreramerino/GearmanBundle/lib

This error doesn't seem to occur when installing other packages. Will provide more details as I have it.

I think client callback are missing..
$gmc->setCreatedCallback("reverse_created");
$gmc->setDataCallback("reverse_data");
$gmc->setStatusCallback("reverse_status");
$gmc->setCompleteCallback("reverse_complete");
$gmc->setFailCallback("reverse_fail");

if not then how can set them ?

Hi,
I'd like to get a feedback from a worker during a long job.

In client I have (in a command with service container)

$dispatcher = $this->getContainer()->get('event_dispatcher');
$dispatcher->addListener("gearman.client.status", array($this, 'onStatus'));

$this->getContainer()->get('gearman')->doJob(...);

in worker

$job->sendData("data: ciao");

The event seems never triggered. Is this approach correct?

Thanks, regards

This bundle should subscribe to tags

• kernel.cache_clearer
• kernel.cache_warmer

... to clear bundle configuration.

Currently there is no way to "reuse" configuration. For example, in a test.yml file, this does not work:

imports:                                                                        
    - { resource: dev.yml }

... and then one could override as-needed for the test environment.

Ideally, configuration would be exposed as described in http://symfony.com/doc/master/cookbook/bundles/extension.html

This way we could use config.yml, config_dev.yml, config_test.yml, etc., with a gearman namespace.

Current Scenario : Workers needs to fetch server parameter from config.yml by default instead of localhost:4730 as default server. If server parameter are not available in config.yml then it needs to be set by default to localhost:4730.

Benefit : If client and worker wants to connect to same server (my scenario and may be most of the time) defined by config.yml then there is no need to specify it's details again in worker annotations.
Otherwise, each time while updating job server, we need to take care of both configuration (host:port config setting should be not duplicated throughout application). If forgotten, will break system.

From below code snippet , it seems that default server has been set to localhost:4730 , not the one in config.yml.

/**
* Adds into worker all defined Servers.
* If any is defined, performs default method
*
* @param \GearmanWorker $gmworker Worker to perform configuration
* @param array $servers Servers array
*/
private function addServers(\GearmanWorker $gmworker, Array $servers)
{
    if (!empty($servers)) {

        foreach ($servers as $server) {

            $gmworker->addServer($server['host'], $server['port']);
        }
    } else {
        $gmworker->addServer();
    }
}

Expected :

1. If specified, set server as specified in annotation.
2. If specified, set server as specified in config.yml. (satisfies my issue.)
3. Otherwise, set server by default to localhost:4730.

I hope, It make sense.

Kindly revert APAS.

Thanks,

Hi mmoreram,

It is good to have your bundle for gearman.
But i am little disappointed that, you closed my open issue #40 before getting confirmation of solved from my side.

Once again i reversed engineered your code and found that,

As per your Documentation configure should have

    namespace: Mmoreramerino\TestBundle

But for your code to work it should be,

    name: MmoreramerinoTestBundle

Also make sure to solved open issue before closing them. So that developer will get proper solution.

I'm getting this error when I attempt to php app/console cache:clear --env=prod. I'm using Symfony 2.1 master and development branch of GearmanBundle.

The following command returns an exception when configuration is different than the test worker. I'm assuming it is a problem with the autoloading.

php app/console gearman:worker:list

Results in:

Fatal error: Class 'PHPUnit_Framework_TestCase' not found in /var/www/project/src/vendor/symfony/src/Symfony/Bundle/FrameworkBundle/Test/WebTestCase.php on line 24

For example, in app/config/gearman/dev.yml the following outputs worker information:

bundles:
    GearmanBundle:                                                                    
        namespace: Mmoreramerino\GearmanBundle                          
        active: true

However, putting the same testWorker.php in a different bundle under a Workers folder, the gearman:worker:list command produces the exception. For example:

bundles:
    GearmanBundle:                                                                    
        namespace: Mmoreramerino\GearmanBundle                          
        active: false
    ProjectBundle:                                                                 
        namespace: MyProject\Bundle\MyBundle                 
        active: true

Anyone with arguments against?

When setting the number of iterations for a job to 1, there will be 2 job executions.
This is the same for 2 iterations, which will result in 3 job executions.

Hi,
Can I move app/config/gearman/dev.yml into my customed bundle configuration directory
like MyBundle/Resources/config/gearman.yml, or move it into sf2 configuration file like app/config_dev.yml, app/config_prod.yml?
Would GearmanBundle support this function?

Thanks.

Kindly correct spelling mistake,

ADDTASKNACKGROUND => ADDTASKBACKGROUND

Gearmanclient.php

 public function addTaskBackground($name, $params =array(), &$context = null, $unique = null)
     {
         $this->enqueueTask($name, $params, $context, $unique, GearmanMethods::GEARMAN_METHOD_ADDTASKNACKGROUND);

         return $this;
    }

This method should be able ( overridable )

When I execute the following code:

$Gearman = $this->get('gearman');
$Gearman->doNormalJob('BundleName~test');

My code throws the following exception:

GearmanClient::doNormal() expects parameter 2 to be string, array given

Stack:

#0 [internal function]: Symfony\Component\Debug\ErrorHandler->handle(2, 'GearmanClient::...', '/Users/reneters...', 178, Array)
#1 /Users/reneterstegen/Sites/core.xxx.nl/vendor/mmoreram/gearman-bundle/Mmoreram/GearmanBundle/Service/GearmanClient.php(178): GearmanClient->doNormal('XXXBundleCoreBu...', Array, NULL)
#2 /Users/reneterstegen/Sites/core.xxx.nl/vendor/mmoreram/gearman-bundle/Mmoreram/GearmanBundle/Service/GearmanClient.php(153): Mmoreram\GearmanBundle\Service\GearmanClient->doEnqueue(Array, Array, 'doNormal', NULL)
#3 /Users/reneterstegen/Sites/core.xxx.nl/vendor/mmoreram/gearman-bundle/Mmoreram/GearmanBundle/Service/GearmanClient.php(266): Mmoreram\GearmanBundle\Service\GearmanClient->enqueue('XXXBundleCoreBu...', Array, 'doNormal', NULL)
#4 /Users/reneterstegen/Sites/core.xxx.nl/src/XXX/Bundle/CoreBundle/Controller/TestController.php(32): Mmoreram\GearmanBundle\Service\GearmanClient->doNormalJob('XXXBundleCoreBu...')
#5 [internal function]: XXX\Bundle\CoreBundle\Controller\TestController->testAction()
#6 /Users/reneterstegen/Sites/core.xxx.nl/app/bootstrap.php.cache(2815): call_user_func_array(Array, Array)
#7 /Users/reneterstegen/Sites/core.xxx.nl/app/bootstrap.php.cache(2789): Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object(Symfony\Component\HttpFoundation\Request), 1)
#8 /Users/reneterstegen/Sites/core.xxx.nl/app/bootstrap.php.cache(2918): Symfony\Component\HttpKernel\HttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true)
#9 /Users/reneterstegen/Sites/core.xxx.nl/app/bootstrap.php.cache(2220): Symfony\Component\HttpKernel\DependencyInjection\ContainerAwareHttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true)
#10 /Users/reneterstegen/Sites/core.xxx.nl/web/app_dev.php(19): Symfony\Component\HttpKernel\Kernel->handle(Object(Symfony\Component\HttpFoundation\Request))
#11 {main}

When I go down to the code the exception is thrown in this function:

/**
 * Execute a GearmanClient call given a worker, params and a method.
 *
 * If he GarmanClient call is asyncronous, result value will be a handler.
 * Otherwise, will return job result.
 *
 * @param array  $worker Worker definition
 * @param mixed  $params Parameters to send to job
 * @param string $method Method to execute
 * @param string $unique A unique ID used to identify a particular task
 *
 * @return mixed  Return result of the GearmanClient call
 */
private function doEnqueue(Array $worker, $params = '', $method = null, $unique = null)
{
    $gearmanClient = new \GearmanClient();
    $this->assignServers($gearmanClient);

    return $gearmanClient->$method($worker['job']['realCallableName'], $params, $unique);
}

Caused by:

public function doNormalJob($name, $params = array(), $unique = null)
{

    return $this->enqueue($name, $params, GearmanMethods::GEARMAN_METHOD_DONORMAL, $unique);
}

Here the default value for $params = array(). In the rest of the chain this parameter is not altered so an array will be given to de doNormal method.

This looks like a bug.

Thanks in advance!

Hello,

Is there a way to get the queue length?
Indeed, I run the method doBackgroundJob when I want to execute an action but I got a problem if the worker isn't launched : my queue increases without any reason. So I'd like to know the queue length in order to avoid useless doBackgroundJob.

Thanks for the answer

Hi mmoreram,

For below setting,

     type: file_system

After running any command second time getting error,

PHP Fatal error: Call to undefined method Doctrine\Common\Cache\FilesystemCache::get() in /var/www/prototype/Symfony/vendor/mmoreram/gearman-bundle/Mmoreram/GearmanBundle/Service/GearmanCacheWrapper.php on line 150

but for ,

    type: array

working fine.

any idea ?

[LogicException]
You must call one of in() or append() methods before iterating over a Finder.

Hi,
Provided documentation link is showing 404.
http://mmoreramerino.github.com/GearmanBundle/annotations.html
is it changed? Kindly, let me know.

Getting this when calling a job with Gearmand 0.34

Deprecated: GearmanClient::do() [gearmanclient.do]: Use GearmanClient::doNormal() in /vhosts/domains/cardsinthepost.com/public/current/vendor/bundles/Mmoreramerino/GearmanBundle/lib/Mmoreramerino/GearmanBundle/Service/GearmanClient.php line 138

see https://scrutinizer-ci.com/g/mmoreram/GearmanBundle/inspections/0f2a2e9e-9950-4b9e-8cfd-cd007fd7ebf0/issues/files/Service/GearmanCallbacks.php?status=existing#inspectioncomment-21767186

The way of reducing this issue is

• Creating an abstract AbstractGearmanCallbacks with the EventDispatcher injection.
• Creating many different GearmanCallbacks, grouped by meaning.

Convert all available Task Callbacks to valid Symfony2 kernel events

Generators\UniqueJobIdentifierGenerator

instead of

Generator\UniqueJobIdentifierGenerator

(no s with Generator)

I'm trying out the following annotation

@gearman\Work(name = "TariochEveapiFetcherEveWorker", service = "tarioch.eveapi.worker.eve")

but an worker:list returns

@Worker:  Tarioch\EveapiFetcherBundle\Component\Worker\EveWorker
callablename:  TariochEveapiFetcherBundleComponentWorkerTariochEveapiFetcherEveWorker

Shouldn't my name be taken instead of the auto generated one from the path?

By default in GearmanBundle generate_unique_key set true. And when I execute next code my worker doesn't get all 500 jobs:

$gearman = $this->getContainer()->get('gearman');
for ($i = 0; $i < 500; $i++) {
    $gearman->doHighBackgroundJob('TestTestBundleWorkersWorker~testB', 'value1');
}

When I set generate_unique_key: false in config everything work fine.

Whenever I get the gearman service it gave me a:

[ReflectionException]
Class does not exist

I got this because of the GearmanCacheWrapper which finds all the files in my bundle, and then checks for the class namespaces, even of non-php files.

lines 268 and 269 of GearmanCacheWrapper.php:
$classNamespace = $this->getFileClassNamespace($file->getRealpath());
$reflClass = new ReflectionClass($classNamespace);

$classNamespace was empty for alot of files and it created this exception.
Quick fix for me was to add:
if( !$classNamespace )
continue;

before the creation of the ReflectionClass.

Any idea how to fix this cleaner as I do not want to edit the vendor file?

Hi,

We are facing some problem with your updated bundle logic.

Scenario -

We want to process on-demand thousand jobs. We want to create workers on-demand which will be alive once started on it own ( not with help of any Supervisord). Then we will terminate them after completing all jobs. ( We will manually run cron to kill them.)

In you previous bundle when we used to set iteration = 0, worker used to be stay alive till get killed manually.

But now for iteration = 0, after processing one job it is shutting down.

Things not possible,

1. As we don't know exact no of jobs , cannot hard code value in iteration annotation to let worker/job know exact it's iteration.

2. If we set iteration = 0 and put it under Supervisord then after completing one job worker will shutdown and Supervisord will try to restart them. So worker can't be killed.

But we don't want multiple workers running in background after all completing their jobs.We will manually run cron to kill them.

Kind integrate your previous logic which was good, flexible and was not keeping any Supervisord kind of dependency.

$alive = false;
if(iteration == 0){
$alive = true;
}

while ($gearmanWorker->work()) {
if ($gearmanWorker->returnCode() != GEARMAN_SUCCESS) {
break;
}
if($alive === false){
if ($iterations-- <= 0) {
break;
}
}
}

Kind revert ASAP ..Thank you.

Right now one place I am using this has multiple environments on the same server, ie dev and staging. I'm not sure the best way to differentiate workers to pull only live jobs vs dev jobs. I'm trying to avoid running multiple instances of gearmand at the moment.

What if you could specify some sort of prefix which all workers/clients could automatically add to their calls? This would let you basically prefix all staging jobs with staging_Foo and dev with dev_Foo etc. Am not sure if this is the best way, and am open to suggestions for how to do it better.

Thanks, and awesome bundle.

One of last Doctrine2 changes make this bundle crash.

PHP Catchable fatal error: Argument 1 passed to Mmoreram\GearmanBundle\Service\GearmanCallbacks::__construct() must be an instance of Symfony\Component\EventDispatcher\EventDispatcher, instance of Symfony\Component\HttpKernel\Debug\TraceableEventDispatcher given, called in /var/www/projects/instantluxe.parsing/app/cache/dev/appDevDebugProjectContainer.php on line 1422 and defined in /var/www/projects/GearmanBundle/Service/GearmanCallbacks.php on line 46

Type Hinting should always be with an interface

Hi,
Below is my gearman client related code,
I wanna ask what's the format of the variable $job (the first argument of doJob function),
thanks.

$gmclient = $this->getContainer()->get('gearman');
$gmclient->setServer("172.16.6.1");
$record = array("msg" => "ok");
$gmclient->doJob($job, $record);

Could anyone please test connection to different gearman server rather than default?
For example i have config:

defaults:
    servers:
        cache:
            hostname: 192.168.0.223
            port: 4730

But client always tries to connect 127.0.0.1

$gearman = $this->getContainer()->get('gearman');

I know the problem is not in the config itself, because when i dump:

$gearman->getSettings()

I got all the server config correctly, so right now my temporary fix i could think of to get things work is:

$gearman = $this->getContainer()->get('gearman');
$settings = $gearman->getSettings()['defaults']['servers']['cache'];
$gearman->setServer($settings['hostname']);
$result = $gearman->doBackgroundJob....

Thank you.

Enable profiler data-collector for Gearman client calls.

Could be nice to enable/disable this options through a parameter boolean. By default should be true.

Also must be documented.

Create a set of tools for easy Gearman metrics. An easy way to check any defined Queue values, jobs and some other stats.

Gearman users, what do you think about this idea?

Please, tell me your needs!