stil / curl-easy Goto Github PK
View Code? Open in Web Editor NEWcURL wrapper for PHP. Supports parallel and non-blocking requests. For high speed crawling, see stil/curl-robot
License: MIT License
cURL wrapper for PHP. Supports parallel and non-blocking requests. For high speed crawling, see stil/curl-robot
License: MIT License
<?php
use cURL\Request;
use cURL\RequestsQueue;
use cUrl\Event;
$request = new Request('http://foo.bar/');
$request->getOptions()
->set(CURLOPT_TIMEOUT, 5)
->set(CURLOPT_RETURNTRANSFER, true);
$request->addListener('complete', function (Event $event) {
$response = $event->response;
$feed = json_decode($response->getContent(), true);
echo $feed['entry']['title']['$t'];
});
$queue = new RequestsQueue();
$queue->attach($request);
$queue->send();
You would expect an complete
event to be triggered, which isn't true.
Example:
$request = new \cURL\Request(....);
// Specially set the variable $logout_url to empty.
$logout_url = '';
$request->getOptions()
->set(CURLOPT_URL, $logout_url)
->set(CURLOPT_COOKIEJAR, $cookie_jar);
// Execute queue
while ($queue->socketPerform()) {
$queue->socketSelect();
}
As a result, we obtain an infinite loop running.
Problem in line code: https://github.com/stil/curl-easy/blob/master/src/cURL/RequestsQueue.php#L202
this is bugging code.
The problem described by link http://stackoverflow.com/questions/23612881/detect-curl-timeout-in-php
Solution:
//$runningBefore = $this->runningCount;
do {
$mrc = curl_multi_exec($this->mh, $this->runningCount);
$this->read();
} while ($mrc === CURLM_CALL_MULTI_PERFORM || $this->runningCount);
/*
$runningAfter = $this->runningCount;
if ($runningAfter < $runningBefore) {
$this->read();
}
*/
I've camed out with this code: https://pastebin.com/TC288kdq.
But, obviusly, it doesn't work.
How can I do that?
Hello,
Using curl-easy to download json data from a restfull service, the charset was encoded in another type different of my petition (UTF-8), and this is a huge problem because I need to use the accent marks (like á, é, í, ó, ú, etc.).
Could your library solve this problem?
Reggards!
My script add 3 request to queue,
but sometime i receive all 3 response, another time i only receive 1 response, another time nothing receive. 👎
`/** @var goi sang ghi chuong trinh $record */
$queue = new RequestsQueue();
$queue->getDefaultOptions()
->set(CURLOPT_TIMEOUT, 5)
->set(CURLOPT_RETURNTRANSFER, true);
$queue->addListener('complete', function (\cURL\Event $event) {
$response = $event->response;
$json = $response->getContent(); // Returns content of response
$feed = json_decode($json, true);
echo '<pre>Response';
print_r($feed);
echo '</pre>';
});
foreach($liveChannelSchedule as $record){
Yii::error($record->title,'datvt');
$request = new Request('http://api.xx.vn/record/update?id='.$record->id);
$queue->attach($request);
/*$record->uuid = rand(100,99999);
$record->recorded_time = new Expression("NOW()");
$record->save(false);*/
echo 'Record: '.$record->id.' - '.$record->title.PHP_EOL;
}
// Execute queue
while ($queue->socketPerform()) {
echo '*';
$queue->socketSelect();
}`
and response run 1th:
Record: 330 - Phim tài liệu Record: 902 - PHIM TRUYỀN HÌNH: Tình người xứ hoa - Tập 12 Record: 587 - 113 online (2) *************<pre>Response</pre><pre>Response</pre><pre>Response</pre>
run again:
Record: 330 - Phim tài liệu Record: 902 - PHIM TRUYỀN HÌNH: Tình người xứ hoa - Tập 12 Record: 587 - 113 online (2) *************<pre>ResponseArray ( [error] => 0 [uuid] => 8520 [id] => 902 ) </pre>****<pre>Response</pre><pre>Response</pre>
I have created a function like this
protected function curl_get_asyn($urls) {
$result = [];
$queue = new RequestsQueue();
$queue->getDefaultOptions()
->set(CURLOPT_TIMEOUT, 5)
->set(CURLOPT_RETURNTRANSFER, true);
// Set function to execute when request is complete.
$queue->addListener('complete', function (\cURL\Event $event) {
$response = $event->response;
$result[] = [
'url' => $event->request->_url,
'response' => $response->getContent()
];
});
foreach ($urls as $key => $url) {
$request = new \cURL\Request($url);
$request->_url = $url;
// Add request to queue
$queue->attach($request);
}
// Execute queue
$timeStart = microtime(true);
while ($queue->socketPerform()) {
usleep(1000);
}
return $result;
}
return $result is always an empty array because $result array inside listener is different from the one which is returned at the last. How can I achieve this ?
It's a very popular problem - how to know which response is which. They may appear in random order, so we need to add a feature allowing to associate responses with their starting parameters.
the composer autoload is use too much memories
request sort by XHProf Incl.MemUse(bytes), as you see the top 10 memories usage list composer use too much.
main()
spl_autoload_call
???_op
Composer\Autoload\ClassLoader::loadClass
Composer\Autoload\includeFile
cURL\RequestsQueue::send
cURL\RequestsQueue::socketPerform
spl_autoload_call@1
ComposerAutoloaderInit3163d95abb499df82cfeb181810bb3d1::getLoader
Composer\Autoload\ClassLoader::loadClass@1
Composer\Autoload\includeFile@1
cURL\RequestsQueue::read
how to load curl-easy without composer autoload?
Make it possible to queue requests with arguments passed though CURLOPT_POSTFIELDS. This would allow packages, e.g., XML, to be queued and sent as a HTTP POST.
Hi;
Can you add Licence?
$queue = new \cURL\RequestsQueue;
$queue->addListener('complete', function (\cURL\Event $event) {
// do some stuff like add to array
});
foreach($buildings as $building){
$request = new \cURL\Request();
$request->_myId = $i;
$postdata="id= $building->id";
// add options to request
$opts = new \cURL\Options;
$opts->set(CURLOPT_URL, 'example,');
$opts->set(CURLOPT_POSTFIELDS, $postdata);
$request->setOptions($opts);
$queue->attach($request);
}
$result = $queue->send();
how can i know when the queue is done with everything ?
// Set function to be executed when request will be completed
$queue->addListener('complete', function (\cURL\Event $event) {
$response = $event->response;
$json = $response->getContent(); // Returns content of response
$feed = json_decode($json, true);
echo $feed['entry']['title']['$t'] . "\n";
});
The event object of the EventDispatcher dont have any response property anymore, how to get the response then?
Thanks for making a great cURL library!
I think the $queue->send() command seems to return true whether or not individual requests succeed. Is this expected behaviour?
Dear All ,
I know that I can set the proxy with
$queue = new \cURL\RequestsQueue;
$queue
->getDefaultOptions()
->set(CURLOPT_RETURNTRANSFER, true)
->set(CURLOPT_PROXY, $proxy)
->set(CURLOPT_FRESH_CONNECT,true);
But this queue setting will be used with each request.
I need to use a different proxy with each request, how can I setup such?
Is there any way to use the lib as such it does not kill active keep-alive connections, so that it sends multiple requests over the same connection?
RequestsQueue should provide feature of limiting total amount of requests processed at once.
hi Mr.Stil,
it's the Non-blocking requests in parallel
compare to others methods Non-blocking requests in parallel
is more fast and better performance?
https://github.com/stil/curl-easy#non-blocking-requests-in-parallel
It would be great if PHP 5.4 support still could given, we use it in a open source project which runs on severall 5.4 Machines .
In PHP versions 5.3.10+ curl_multi_select changed its behaviour.
Probably bug: https://bugs.php.net/bug.php?id=63411
It causes problem with cURL\RequestsQueue::send()
which is not working now
Temporary workaround: instead of
$queue->send();
You should do:
while ($queue->socketPerform()) {
$queue->socketSelect();
}
Hi, is this still maintained? I would need a symfony dependencies upgrade...
hi, what correct way to show progress while proccess running (non blocking)
I have tried this
<?php
require"vendor/autoload.php";
file_put_contents( 'progress.txt', '' );
$request = new \cURL\Request('http://ftp.free.org/mirrors/releases.ubuntu-fr.org/11.04/ubuntu-11.04-desktop-i386-fr.iso');
$request->getOptions()
->set(CURLOPT_NOPROGRESS,false)
->set(CURLOPT_PROGRESSFUNCTION,'progressCallback')
->set(CURLOPT_RETURNTRANSFER, true);
$request->addListener('complete', function (\cURL\Event $event) {
$response = $event->response;
$data = $response->getContent();
file_put_contents("test.iso",$data);
echo"complete";
});
function progressCallback( $download_size, $downloaded_size, $upload_size, $uploaded_size )
{
static $previousProgress = 0;
if ( $download_size == 0 )
$progress = 0;
else
$progress = round( $downloaded_size * 100 / $download_size );
if ( $progress > $previousProgress)
{
$previousProgress = $progress;
$fp = fopen( 'progress.txt', 'a' );
fputs( $fp, "$progress\n" );
fclose( $fp );
}
}
while ($request->socketPerform()) {
$array=file("progress.txt");
$count=count($array)-1;
echo $array[$count];
$request->socketSelect();
}
Hi, I would like to use your library, but it does not have a version number - I can only install it via composer if I use * as the version. My employer does not like that (if a backward incompatible update is made to the library, it will automatically be pulled in on the next composer update, even if it has a new major version number). It may well be that you don't intend to make any updates, but as a matter of good practice, a tag with a version number would be appreciated. Thanks!
Hi,
how can I send in a single request (only one url) post json_encode two or more arrays, for example:
url: my.urlapi.com
post_fields = array (
array (name => Jhon, age => 30),
array (name => Doe, age => 25),
and more 100 arrays...
);
The destination server only allows 1 entry per request. With your library, is it possible to apply more than one entry in the same request?
Thanks!
I have tried to include this library using composer but still i am getting class not found error Can any one tell me how can i fix this one
for including library i used :- require_once 'vendor/autoload.php';
I am busy with creating a server that reads info from mqtt and needs to process it (post it to another url). Therefor I found this script.
What I need to do is set up the curl-easy queue, and when I have on MQTT a new message, there should be added a requests.
What I Tried now is on starting of the MQTT handler:
$queue = new \cURL\RequestsQueue;
$queue
->getDefaultOptions()
->set(CURLOPT_RETURNTRANSFER, true);
Then every time there is a new requests I add it like this
$req = new \cURL\Request('<url>'); $req->getOptions()->set(CURLOPT_POSTFIELDS, $data); $requests[] = $req;
And after every message I call
$queue->attach(array_pop($requests)); $queue->send();
But is that the correct way? Because now I think that the queue is executed multiple times.
In fact the final result should be:
and that in the background the requests are done (with a maximum nr of requests at a certain time would be very nice).
Thanks
Is it possible to add to the Request class a variable where you can get the count of the queue elements which is internally used?
I want to use it to avoid the string comparision on my usage:
foreach($this->requests as $key => $request) {
/** @var Request $request */
try {
if ($request->socketPerform()) {
$request->socketSelect();
}
} catch(Exception $e) {
if ($e->getMessage() == "Cannot perform if there are no requests in queue.") {
unset($this->requests[$key]);
} else {
throw $e;
}
}
}
Is there a way to send 2 requests to the same url in parallel, but on one request a cookie is set to "Cookie: user=1" and on the 2nd request it's set to "Cookie: user=2" for example?
Thanks!
Hi,
I am having trouble getting error code and message. $response->getError();
returns null.
public function curl_req($path, $data = array())
{
$request = new \cURL\Request($this->_url . $path);
$request->getOptions()
->set(CURLOPT_TIMEOUT, 5)
->set(CURLOPT_RETURNTRANSFER, true)
->set(CURLOPT_HEADER,true)
->set(CURLOPT_VERBOSE,true)
->set(CURLINFO_HEADER_OUT,true)
->set(CURLOPT_USERAGENT,'curl/7.54.0')
->set(CURLOPT_HTTPHEADER, $this->getHeader());
if( !empty($data) ) {
$request->getOptions()
->set(CURLOPT_POSTFIELDS, json_encode($data))
->set(CURLOPT_SAFE_UPLOAD, true);
}
if ($response->hasError()) {
$error = $response->getError();
$e = 'Error code: ' . $error->getCode() . 'Message: "' . $error->getMessage() . '"' . "\n";
}
$this->getLogger()->info('Connect to server: '.$e. ' try: '.$try);
$feed = json_decode($response->getContent(), true);
return $feed;
}
curl verbose:
< HTTP/1.1 401 Unauthorized
< Server: nginx/1.10.0 (Ubuntu)
< Content-Type: application/json
< Transfer-Encoding: chunked
< Cache-Control: no-cache
< WWW-Authenticate: WSSE realm="Secured API", profile="UsernameToken"
< Date: Tue, 12 Sep 2017 12:09:56 GMT
< Set-Cookie: SERVERID=web1; path=/
<
Mr.Stil,
in my crawler i have include a html tidy function : htmLawed.php
in the queue Listener how to injection this htmLawed($contents, $config) function?
include 'htmLawed.php';
$config = array('safe'=>1);
$queue->addListener('complete', function (\cURL\Event $event) use (&$data)
{
$response = $event->response;
$dom = new DOMDocument();
$dom->preserveWhiteSpace = false;
$dom->loadHTML($response->getContent());
$xpath = new DomXPath($dom);
$query = $xpath->query($data['xpath']);
// tidy the html
$content = htmLawed($dom->saveXML($query->item(0)), $config);
// htmLawed() function not in queue
}
I'm trying to use this class to make hundreds of thousands of requests over time (8 at a time) in a long running process and I found that there are memory leaks with this system (or the Symphony Event Dispatcher).
Every additional request seems to add 30kb to the memory usage.
Ok, so when I use the normal cURL, if I get a web like a string, later I can use it with DOMDocument, but in this case, the string explote. I don't know why. I read the documentation and I found nothing, maybe it's because is late but I think it should work.
$request = new \cURL\Request(); $request->getOptions() ->set(CURLOPT_TIMEOUT, 5) ->set(CURLOPT_RETURNTRANSFER, true) ->set(CURLOPT_URL, 'https://www.google.com/'); $response = $request->send(); $feed = $response->getContent(); $dom = new DOMDocument(); $dom->formatOutput = true; $dom->loadHTML($feed);
When I load the string $feed, it explotes. I tried to echo the string $feed and it is the website I try to get.
Hey all,
I believe it might be related to #14
Anyhow, I will ask :)
Here is the idea.
$rows = [1 => 'url1', 5 => 'url2', ... ];
$requests = [];
foreach ($rows as $id => $url) {
$requests[$id] = new \cURL\Request($url);
}
Is there any way to get the key ID?
$queue->addListener('complete', function (\cURL\Event $event) use (&$requests) {
$response = $event->response;
$content = $response->getContent(); // Returns content of response
// TODO: get id here
if ($next = array_pop($requests)) {
$event->queue->attach($next);
}
});
$queue->attach(array_pop($requests));
$queue->attach(array_pop($requests));
$queue->send();
Thanks in advance.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.