Code Monkey home page Code Monkey logo

snidel's Introduction

Snidel

A multi-process container. Snidel makes it easier for all PHP developers to work with parallel processing without any extensions.

Latest Stable Version License Tests Scrutinizer Code Quality Coverage Status Minimum PHP Version

Please consider donating to this project's author, Akihito Nakano, to show your ❤️ and support.

Sponsor @ackintosh on GitHub Sponsors

What Snidel solves?

(en)

Not a few people, start their programming carrier with PHP, and go on. Parallel processing, they are not familiar with it, and may be a hurdle for them.

Or else, people who limited to develop with a language that is not PHP (e.g. A language that has superior feature for parallel processing). (It's me in past.)

To make parallel processing more easily and instinctively to them to use, I started developing Snidel.

Snidel can be one of your options when you are considering "How to do it parallelly?". It's an honer for me.

(ja)

PHPでプログラミングに入門して、PHPでキャリアを積み重ねているプログラマがある程度いると思います(私がそうです)。そのような方にとって並列処理は馴染みがなかったり、敷居の高さを感じるのではないでしょうか。

或いは諸事情によりPHP以外の言語(例えば、並列処理のための優れた機構を持った言語)を利用する事を制限されている中で開発を進めなければならない状況にある方もいらっしゃるでしょう(以前の私がそうでした)。

そのような方が、手軽に・直感的に並列処理を使って問題解決できることを目的として Snidel の開発をはじめました。

"この処理を並列に実行したいんだけどどうしよう?" といった場合に Snidel がみなさんの選択肢のひとつになれたら幸いです。

Installing Snidel via Composer

$ composer require ackintosh/snidel:~0.11.0

Architecture

Master - Worker Architecture

Benefits

It is also possible parallel processing via build-in functions (e.g. exec):

initialize_data_required_for_the_slow_jobs();

exec('php slow_job1.php &');
exec('php slow_job2.php &');

For the developers who feels "pain" with the above, Snidel can provides pretty good experience and will streamline their PHP programming.

We will walk through usage to show how Snidel melt parallel processing into your programming. The experience using Snidel should resolve your pain. Let's get started!

Usage

Basic Usage

<?php
use Ackintosh\Snidel;

$f = function ($s) {
    sleep(3);
    echo 'echo: ' . $s;
    return 'return: ' . $s;
};

$s = time();
$snidel = new Snidel();
$snidel->process($f, ['foo']);
$snidel->process($f, ['bar']);
$snidel->process($f, ['baz']);

// `Snidel::results()` returns `\Generator`
foreach ($snidel->results() as $r) {
    // string(9) "echo: foo"
    var_dump($r->getOutput());
    // string(11) "return: foo"
    var_dump($r->getReturn());
}

// If you don't need the results, let's use `Snidel::wait()` instead of `Snidel::results()`
// $snidel->wait();

echo (time() - $s) . 'sec elapsed' . PHP_EOL;
// 3sec elapsed.

Constructor parameters

All parameters are optional.

new Snidel([
    'concurrency' => 3,
    // Please refer to `Logging`
    'logger' => $monolog,
    // Please refer to `Using custom queue`
    'driver' => $driver,
    // a polling duration(in seconds) of queueing
    'pollingDuration' => 1,
]);

Same arguments as call_user_func_array

// multiple arguments
$snidel->process($f, ['arg1', 'arg2']);

// global function
$snidel->process('myfunction');

// instance method
$snidel->process([$instance, 'method']);

Tagging the task

$f = function ($arg) {
    return $arg;
};

$snidel->process($f, 'arg-A_tag1', 'tag1');
$snidel->process($f, 'arg-B_tag1', 'tag1');
$snidel->process($f, 'arg_tag2', 'tag2');

foreach ($snidel->results as $r) {
    // `Task::getTag()` returns the tag passed as 3rd parameter of `Snidel::process()`
    switch ($r->getTask()->getTag()) {
        case 'tag1':
            $r->getReturn(); // arg-A_tag1 | arg-B_tag1
            break;
        case 'tag2':
            $r->getReturn(); // arg_tag2
            break;
        default:
            $r->getReturn();
            break;
    }
}

Logging

Snidel supports logging with logger which implements PSR-3: Logger Interface.

// e.g. MonoLog
use Monolog\Formatter\LineFormatter;
use Monolog\Handler\StreamHandler;
use Monolog\Logger;

$monolog = new Logger('sample');
$stream = new StreamHandler('php://stdout', Logger::DEBUG);
$stream->setFormatter(new LineFormatter("%datetime% > %level_name% > %message% %context%\n"));
$monolog->pushHandler($stream);

$snidel = new Snidel(['logger' => $monolog]);
$snidel->process($f);

// 2017-03-22 13:13:43 > DEBUG > forked worker. pid: 60018 {"role":"master","pid":60017}
// 2017-03-22 13:13:43 > DEBUG > forked worker. pid: 60019 {"role":"master","pid":60017}
// 2017-03-22 13:13:43 > DEBUG > has forked. pid: 60018 {"role":"worker","pid":60018}
// 2017-03-22 13:13:43 > DEBUG > has forked. pid: 60019 {"role":"worker","pid":60019}
// 2017-03-22 13:13:44 > DEBUG > ----> started the function. {"role":"worker","pid":60018}
// 2017-03-22 13:13:44 > DEBUG > ----> started the function. {"role":"worker","pid":60019}
// ...

Error informations of children

$snidel->process(function ($arg1, $arg2) {
    exit(1);
}, ['foo', 'bar']);
$snidel->get();

var_dump($snidel->getError());
// class Ackintosh\Snidel\Error#4244 (1) {
// ...
// }

foreach ($snidel->getError() as $pid => $e) {
    var_dump($pid, $e);
}
// int(51813)
// array(5) {
//   'status' =>  int(256)
//   'message' => string(50) "an error has occurred in child process.
//   'callable' => string(9) "*Closure*"
//   'args' =>
//     array(2) {
//       [0] => string(3) "foo"
//       [1] => string(3) "bar"
//     }
//   'return' => NULL
//   }
// }

Using custom queue

Snidel depends on Bernard as a queue abstraction layer. Bernard is a multi-backend PHP library for creating background jobs for later processing.
By default Snidel builds the flatfile driver, but from a race condition perspective, we recommend using a more reliable queue in production.

Amazon SQS
$connection = Aws\Sqs\SqsClient::factory([
    'key'    => 'your-aws-access-key',
    'secret' => 'your-aws-secret-key',
    'region' => 'the-aws-region-you-choose'
]);
$driver = new Bernard\Driver\SqsDriver($connection);

new Snidel([
    'driver' => $driver,
]);

For details on the driver, please see here.

Articles

Here are articles that introducing Snidel. Thank you!

Requirements

Version Guidance

Snidel PHP
0.1 ~ 0.8 >= 5.3
0.9 ~ >= 5.6
0.13 >= 7.1

Docker

We suggest you give it a try with Docker as Snidel requires some php extensions shown in Requirements.

Run unit tests in docker container

curl -Ss https://getcomposer.org/installer | php
docker build -t snidel .
docker run --rm -v ${PWD}:/snidel snidel php composer.phar install
docker run --rm -v ${PWD}:/snidel snidel vendor/bin/phpunit

Author

Snidel © ackintosh, Released under the MIT License.
Authored and maintained by ackintosh

GitHub @ackintosh / Twitter @NAKANO_Akihito / Blog (ja)

Blog entries by author about Snidel (ja):

Acknowledgments

Thanks to JetBrains for supporting us with a Free Open Source License.

snidel's People

Contributors

ackintosh avatar atefbb avatar ravelll avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

snidel's Issues

declare tick or

https://github.com/ackintosh/snidel/blob/master/src/Snidel/Fork/Coordinator.php#L152
(From my comment at BernardPHP)
Declare tick should maybe not be used at all, many places on the web it is suggested to use pcntl_signal_dispatch instead.
Inside the drivers, instead of usleep(), could use time_nanosleep, which is interruptable.

pcntl_signal_dispatch();
$nano = time_nanosleep(0, 100000000);
if ($nano === true) {
  // slept
} elseif ($nano === false) {
  // nanosleep fail
  usleep(100000);
} elseif (is_array($nano)) {
  // Interrupted by a signal
  pcntl_signal_dispatch();
  return array(null, null);
}
pcntl_signal_dispatch();

(edit: via GeniusesOfSymfony/WebSocketBundle#125 )

the second argument of the `fork` method.

Warning occurs when omit the second argument of the fork method.

$snidel->fork(function () {
    return 'foo';
});

// PHP Warning:  call_user_func_array() expects parameter 2 to be array, null given in ...

Breaking changes in bernardphp/bernard

Hi!

I've looked into your library and it looks like with the latest changes in master it's going to break. Some massive refactors and rewrites are happening at the moment. Until those are sorted out, you might want to consider locking to one of the alpha versions.

get results with tags

$snidel->fork($func, 'foo', 'tag1');
$snidel->fork($func, 'bar', 'tag2');
$snidel->fork($func, 'baz', 'tag3');

$snidel->join();

echo $snidel->get('tag1');
// foo

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.