Pipelining? about urllib3 HOT 12 CLOSED

@hannosch: You can use pipeline function from this code, use that function with httplib:

As far as I know, pipeline is not as fast as reuse connection like urllib3, so you should just use urllib3 is enough, I think maybe HTTP2.0's pipeline might be faster than reuse connection but not sure at this time.

"""
Using the builtin http client, demonstrate different strategies for multiple requests:
- serial requests with one connection (the baseline)
- pipelining requests with one connection
- conncurrent requests with multiple connections

When reusing connections, each response must be fully read before using the next response.
Generator expressions are returned so requests are sent immediately, and the timing is fair.
"""

from __future__ import print_function
try:
    from http import client
except ImportError:
    import httplib as client


def sequential(host, *requests):
    "Send requests (method, path[, body[, headers]]) and generate responses serially."
    conn = client.HTTPConnection(host)
    return (conn.request(*request) or conn.getresponse() for request in requests)


def pipeline(host, *requests):
    "Pipeline requests (method, path[, body[, headers]]) and generate responses."
    conn = client.HTTPConnection(host)
    responses = []
    for request in requests:
        conn.request(*request)
        responses.append(conn.response_class(conn.sock, method=conn._method))
        conn._HTTPConnection__state = 'Idle'
    return (response.begin() or response for response in responses)


def concurrent(host, *requests):
    "Send requests (method, path[, body[, headers]]) in parallel and generate responses."
    conns = []
    for request in requests:
        conns.append(client.HTTPConnection(host))
        conns[-1].request(*request)
    return (conn.getresponse() for conn in conns)


if __name__ == '__main__':
    import time
    requests = [('GET', '/delay/1')] * 3
    for func in (sequential, pipeline, concurrent):
        print(func.__name__)
        responses = func('httpbin.org', *requests)
        start = time.time()
        for response in responses:
            assert response.status == client.OK and response.read()
            print(time.time() - start)

from urllib3.

@shazow and I have discussed this a few months ago, and determined that there was little benefit.

from urllib3.

@jtolds Hey JT, this is totally the right venue for this, thanks for posting. I don't think we discussed pipelining too much. I think @kennethreitz is referring to our SPDY discussions.

Any idea what the server support for HTTP pipelining is like these days?

Is this something you'd be interested in implementing?

from urllib3.

Seems most browsers don't support pipelining: http://en.wikipedia.org/wiki/HTTP_pipelining#Implementation_in_web_browsers

And proxy support is another thing that will break.

Are we sure this is worth the trouble? If you need this for a specific usecase, I'd be open for a composite implementation which can be used optionally but not enabled by default.

from urllib3.

I'm not sure what the general server support is. I saw a list somewhere but all I can find now is just Wikipedia's entry on pipelining support: http://en.wikipedia.org/wiki/HTTP_pipelining (Implementing pipelining in web servers is a relatively simple matter of making sure that network buffers are not discarded between requests. For that reason, most modern web servers handle pipelining without any problem.)

For my case, since I'm talking to a custom server that I control (this isn't web traffic, this is an application protocol over REST), I'd be okay if the default was not to do pipelining unless the caller manually turns it on.

There is a very good chance I'll be interested in implementing this. I'm currently working on getting a minimum set of functionality done on a product I'm working on - but it's likely I will for sure circle back. It's good to know you're welcoming to patches for this functionality.

If it ends up getting implemented before I get to it, awesome, otherwise, stay tuned. :)

from urllib3.

i think you can send pipelined requests at a time. if the body= of your second request is a pipeline object, it will iteratively call .Request() with a reasonable delay.

from urllib3.

Thanks, @allover, for your snarky reply. I've left an update on #51

from urllib3.

I hope we're past the bickering now. :)

@jtolds you're welcome to try and implement this. If you can find a way to do this that doesn't increase code complexity significantly, I'd love to try and merge it in. Please include tests, too. :)

from urllib3.

no prob - again, not sure i'll get to it in the next week or two, but i'll let you know. :)

thanks again

from urllib3.

Just as a motivation for pipelining: most mobile browsers have it enabled by default, so pretty much any web site has to support it these days. See http://www.blaze.io/mobile/http-pipelining-big-in-mobile/, http://www.mnot.net/blog/2011/08/05/pipeline_now and http://tech.vg.no/2011/12/14/safari-on-ios-5-randomly-switches-images - bottom line: Android, Opera Mobile, Opera Deskop and Safari since iOS 5 all use it, Firefox and Chrome are planning to enable it.

from urllib3.

@Lukasa is working on an HTTP/2 lib which would have a more natural API handling concurrency/pipelining: https://github.com/Lukasa/hyper

I hope to someday use it as urllib3's backend. Closing this until then. :)

from urllib3.

I was able to make it work with the following hack:
Before sending a pipelined request:
conn._HTTPConnection__state = httplib._CS_IDLE
Before getresponse:
conn._HTTPConnection__state = httplib._CS_REQ_SENT
where conn is the httplib connection object.
In my case the requests are identical except for one query variable.
I sent 74 requests in a row. Only then did the getresponse/read for all requests.

from urllib3.