Comments (5)
This is because of #8 (urllib3 sends the entire url in the GET line, instead of just the path). Seems the servers running waptt.com don't like that.
I would suggest using https://github.com/kennethreitz/requests for now, which is built on top of urllib3 but does a lot of extra things for you like strip out the scheme/host from the GET line before sending it.
from urllib3.
I've tested the grequests (Requests + Gevent https://github.com/kennethreitz/grequests) and Urllib3 the performance comparison and concluded that much better gevent + urllib3 performance than grequests ,so I gevent + urllib3
from urllib3.
Interesting. Could you share your benchmark methodology and numbers? I'm curious to see where Requests is slow; I'm sure we can speed it up.
My other suggestion, for now, would be to implement your own PoolManager which removes the scheme+host from the request url before passing it on.
from urllib3.
Requests speed is slower and more error
python gtest.py
by requests: 25.7797329426 seconds
by urllib3: 1.32646393776 seconds
Test code:
import sys
import gevent
from gevent import monkey
gevent.monkey.patch_all(thread=False)
import grequests
import urllib3
http = urllib3.PoolManager()
def call_back(resp):
content = resp.content
def worker(url, use_urllib2=False):
if use_urllib2:
content = http.request('GET', url)
else:
rs = [grequests.get(u) for u in url]
resps = grequests.map(rs)
for resp in resps:
call_back(resp)
urls = ['http://www.baidu.com/']*50
def by_requests():
worker(urls)
def by_urllib2():
jobs = [gevent.spawn(worker, url, True) for url in urls]
gevent.joinall(jobs)
if __name__=='__main__':
from timeit import Timer
t = Timer(stmt="by_requests()", setup="from __main__ import by_requests")
print 'by requests: %s seconds'%t.timeit(number=3)
t = Timer(stmt="by_urllib2()", setup="from __main__ import by_urllib2")
print 'by urllib3: %s seconds'%t.timeit(number=3)
from urllib3.
@kennethreitz, thoughts?
from urllib3.
Related Issues (20)
- Create a workflow (nox?) for testing Emscripten support locally
- Emscripten support emits an InsecureRequestWarning even when using HTTPS
- Path toward testing with a released version of hypercorn? HOT 3
- Handle HTTP/2 informational responses (1xx) HOT 2
- urllib3 2.2 explicitly casts all headers to HTTPHeaderDict HOT 4
- Streaming responses using urllib3 HOT 5
- verbose logging output
- Excess leading path separators causes ConnectionPool.urlopen to parse URL as host & port HOT 1
- ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) HOT 1
- SSL: UNEXPECTED_EOF_WHILE_READING HOT 7
- imprecise types on `urllib3.Retry.new` / `urllib3.Retry.increment` HOT 1
- Investigate CI failures with Python 3.13.0a5 HOT 2
- "unable to get local issuer certificate", even though cURL works with the same website HOT 3
- Unclosed socket warning after HTTP 407 response from HTTP CONNECT proxy HOT 1
- All Retry backoff_factor to optionally start applying from first retry HOT 4
- Retry backoff_factor offset from second retry incorrectly computed HOT 2
- HTTPConnection.request chunked=False doesn't work properly HOT 8
- Need to exception for "SSLEOFError" on python 3.10, 3.11, 3.12 HOT 3
- After upgrading to 2.2.1: 'HTTPResponse' object has no attribute 'json' HOT 1
- Comment typo settimout settimeout HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from urllib3.