jyggen / curl Goto Github PK

Currently Session store the response as an array within itself, and we should look into using Symfony's Response class or something instead. This would give the end-user more control over the data/response and limit the scope of Session to what it was intended for, a wrapper around cURL handlers.

To avoid running out of memory when executing a huge amount of requests we need a way to interact and process the response as soon as the request is executed.

Let's pretend you want to do multiple requests against a JSON API and print out the ID. You've got a dispatcher with attached requests and want to process the data. Currently this is the easiest way to do it:

$dispatcher->execute();

foreach ($requests->get() as $request) {
    print json_decode($request->getResponse()->getContent())->id;
}

Now if this is a huge JSON file (or maybe even a XML file) and you want to print the ID of tons of different resources you'll probably hit PHP memory limit eventually.

Now imagine if you could do this instead:

$dispatcher->execute(function($content) {
    print json_decode($content)->id;
});

To get this working we'll first have to implement #8 and then after each batch of requests we'll pass each request's content to the closure (the HTTP status code might be useful as well).

Will this actually fix the underlying issue or just give us a nice-to-have method? I don't know, I'll have to research it some more so we might have to do some architectural work as well.

Hi!
Sometimes i get response with headers like

Content-Type:
(the value is empty)

In this case i get an error:

Undefined offset: 1

curl/src/Response.php:56
curl/tests/ResponseTest.php:48

Thanks!

I'm pretty sure that this is wrong:

$session->setOption(CURLOPT_INFILESIZE, mb_strlen($data, 'UTF-8'));

Shouldn't that be?

$session->setOption(CURLOPT_INFILESIZE, strlen($data));

Hi,

When doing a post request I want the data to be sent as postfield and not with http_build_query

Example:

This works:

$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, "https://clients6.google.com/rpc");
curl_setopt($curl, CURLOPT_POST, 1);
curl_setopt($curl, CURLOPT_POSTFIELDS, '[{"method":"pos.plusones.get","id":"p","params":{"nolog":true,"id":"' . $url . '","source":"widget","userId":"@viewer","groupId":"@self"},"jsonrpc":"2.0","key":"p","apiVersion":"v1"}]');
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl, CURLOPT_HTTPHEADER, array('Content-type: application/json'));
$curl_results = curl_exec ($curl);
curl_close ($curl);
$json = json_decode($curl_results, true);
var_dump($json);

Now using this library

$requestURI = env('GOOGLE_COUNT_URL') . '?key='. env('GOOGLE_PUBLIC_API_KEY');
$data       = '[{"method":"pos.plusones.get","id":"p","params":{"nolog":true,"id":"' . $url . '","source":"widget","userId":"@viewer","groupId":"@self"},"jsonrpc":"2.0","key":"p","apiVersion":"v1"}]';
$response = Curl::post($url, $data); 

This throws the following exception

ErrorException in Curl.php line 155:
http_build_query(): Parameter 1 expected to be Array or Object. Incorrect value given

When I pass the data as json_decoded array/object I get null as response

$data       = json_decode('[{"method":"pos.plusones.get","id":"p","params":{"nolog":true,"id":"' . $url . '","source":"widget","userId":"@viewer","groupId":"@self"},"jsonrpc":"2.0","key":"p","apiVersion":"v1"}]'); 

curl is a asynchronous client for c, but a concurrency client for php.

The Dispatcher and Response pages result in errors:

http://docs.jyggen.com/curl/dispatcher
http://docs.jyggen.com/curl/response

There's really no logic behind the dispatcher having a getResponses() method, it should keep track of sessions and not their responses!

This will, however, make the API somewhat more complicated. You would first have to call getSessions() on the dispatcher and then for each session you would have to call getResponse().

Hopefully it's worth it to keep the library persistent and clean. In most cases you use the static helpers anyway, and they would do it for you.

It work with files

Hi, tried you library and bare curl.

Bare Curl

for ($page = 1; $page <= 30; $page++)
{
    $url = 'http://example/page/' . $page . '/';
    $curl = curl_init($url);
    curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
    $response = curl_exec($curl);
    if (curl_errno($curl))
    {
        die('<p>Ошибка: ' . curl_error($curl) . '<p>');
    }
    curl_close($curl);
    echo '<p>' . Helpers::memoryUsage() . '</p>';
}

The memory usage is 6-7 mb after the execution.

This library

Option 1

for ($page = 1; $page <= 30; $page++)
{
    $url = 'http://example/page/' . $page . '/';
    $response = \Jyggen\Curl\Curl::get($url);
    unset($response);
    echo '<p>' . Helpers::memoryUsage() . '</p>';
}

Option 2

for ($page = 1; $page <= 30; $page++)
{
    $url = 'http://example/page/' . $page . '/';
    $request = new \Jyggen\Curl\Request($url);
    $request->execute();
    $response = $request->getResponse();
    unset($request);
    unset($response);
    echo '<p>' . Helpers::memoryUsage() . '</p>';
}

Memory usage in both options is 23-24 mb after execution.
It seems like it does not erase from memory, manually triggering Request __destructor() makes memory usage about 18-19 mb, still bad vs bare curl.
Multi curl is also 23-24 mb.

Why ? Am I doing smth wrong ?

Hello!

Can i get RAW data from multiple requests?

$allData=Curl::get(array('www.1page.com/jsonData','www.2page.com/jsonData'));

I need all data in RAW(JSON)

Best regards,
Eugene

Session should most likely be renamed to Request since that's what it actually is, a request.

Thinking about implementing some kind of session support to the library (stuff persists through requests), and this would kind of block that class. Unless, of course, we call that class Request instead (oh the mindfuck).

Session should implement an interface to improve the extensibility of the library. It should be okay to roll your own session handler, since most of the library's core functionality lies within Dispatcher.

Could you please copy them in doc folder in this repository?

Hey guys,

I got a strange behavior when using Jyggen Curl in Laravel to download some JPEGs.
Problem was, that the file got downloaded, but was corrupted (the headers i think)

I took the Basic curl usage, made the same options and request - and it worked!
Had no time for further debugging, but here's what I did:

NON working example:

$fp = fopen ($dirTarget.$filename, 'wb+');
$request = new Request($uriSrc);
$request->setOption(CURLOPT_BINARYTRANSFER, 1);
/* Protected Option. Unable to set...
$request->setOption(CURLOPT_RETURNTRANSFER, 1);*/
$request->setOption(CURLOPT_FILE, $fp);
$request->execute();
fclose($fp);  

WORKING example:

$fp = fopen ($dirTarget.$filename, 'wb+');
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, $uriSrc);
curl_setopt($curl, CURLOPT_BINARYTRANSFER, 1);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl, CURLOPT_FILE, $fp);
$result = curl_exec($curl);
fclose($fp);

Any Ideas?

Regards,
Steffen

'ErrorException' with message 'curl_setopt(): The usage of the @filename API for file uploading is deprecated. Please use the CURLFile class instead'

Currently the Dispatcher class uses method names such as addSession(), clearSessions() and removeSession(). Since the dispatcher is a handler for our sessions it shouldn't confuse anyone if we made them shorter.

This is what I have in mind:

• addSession() -> add()
• clearSessions() -> clear()
• getSessions() -> get()
• removeSession() -> remove()

Also, we could change getResponses() to responses() since it would be awkward to have a get() and a getResponses() method. But if issue #1 is fixed this isn't a problem anyway.

This site: http://docs.jyggen.com/curl

Gives 404

When i call Curl::get() i take:

array (size=1)
  0 => 
    object(jyggen\Curl\Response)[522]

Currently all requests added to the dispatcher are executed at the same time. However, this can put a huge load on, and potentially hurt, both the local and remote server.

This can be fixed by spliting the requests into stacks and then go through each stack and execute those requests. So instead of executing all 1000 requests at the same time we could execute them in batches of say 50.

Hello, I'm trying to make a curl request with your library. Everything works fine except for the fact that I get the header with my response. Since the rest of my response is JSON I can not seem to remove the headers to be able to read my response.

In the Request.php the CURLOPT_HEADER is protected so I can not overwrite it during my curl request. Is there anyway way we can change this with a config file or anything? When I put it on false my curl request works perfectly fine.

Thanks for the help!