spyoungtech / grequests Goto Github PK
View Code? Open in Web Editor NEWRequests + Gevent = <3
Home Page: https://pypi.python.org/pypi/grequests
License: BSD 2-Clause "Simplified" License
Requests + Gevent = <3
Home Page: https://pypi.python.org/pypi/grequests
License: BSD 2-Clause "Simplified" License
Hi. I am using grequests to send a bunch of requests. After about 2000 requests, error below is reported.
error: cannot switch to a different thread (libev) error creating signal/async pipe: Too many open files
Any idea what happened? I use imap with default pool size.
Making requests to https gives me [ grequests.map(rs) gives me ] :
ConnectionError: HTTPSConnectionPool(host='www.foobar.com', port=443): Max retries exceeded with url: /index/xx/82
<Greenlet at 0x3d8a2958: <bound method Request.send of <Request [GET]>>(prefetch=True)> failed with ConnectionError
I would like to be able to add custom data to a request object and get it back from response object. Primarly as a way to distinguish responses from async requests. Something like:
requests = [grequest.get(url, user_info={'some_id': x}) for url, x in ...]
for response in grequests.map(requests):
if (response.user_info['some_id'] == ...) ...
Per the plain-old requests documentation, redirects are followed for all requests except HEAD: http://docs.python-requests.org/en/latest/user/quickstart/#redirection-and-history
However, grequests follows redirects on HEAD unless explicitly told not to:
>>> bad=grequests.map([grequests.head('http://github.com')])[0]
>>> bad.status_code
200
>>> bad.history
(<Response [301]>,)
>>> bad.url
u'https://github.com/'
>>> good=grequests.map([grequests.head('http://github.com', allow
_redirects=False)])[0]
>>> good.status_code
301
>>> good.history
[]
>>> good.url
u'http://github.com/'
Safe mode doesn't work when making async requests. The reason is catch_exceptions_if_in_safe_mode()
decorates request()
which calls Session.request()
which does not call Request.send()
when send_request
is False
which is the case with async requests.
I guess safe mode should guard two phases; one is the creation of a request, and the second one is sending a request.
see kennethreitz/requests#583
I was looking forward to using this awesome package with my app and gevent until I noticed that it accomplishes gevent-compatibility via gevent.monkey.patch_all():
try:
import gevent
from gevent import monkey as curious_george
from gevent.pool import Pool
except ImportError:
raise RuntimeError('Gevent is required for grequests.')
# Monkey-patch.
curious_george.patch_all(thread=False, select=False)
Unfortunately, patch_all() impacts all other packages/modules used by the app, breaking some of them that were not designed for reentrancy. For example, gevent.monkey.patch_all() broke the combination of pymysql/pooled_db (with sporadic, hard-to-reproduce failures); my app has several greenlets that on occasion make pymysql calls; I isolated the problem to gevent's monkey-patching of the socket module in that case.
A local solution involving monkey-patching of just the requests package to use gevent's variants of blocking API's would greatly enhance compatibility of grequests with apps and eliminate unexpected/undesirable side-effects on other packages. For example, I recently monkey-patched HBase/Thrift via the following code snippet with the desired effect on Thrift and no harm to the rest of the app (pymysql, etc.):
# Monkey-patch Thrift's TSocket to use gevent.socket to make our HBase interface
# gevent-friendly
import gevent.socket
from thrift.transport import TSocket
TSocket.socket = gevent.socket
gevent-zeromq is another example of local monkey-patching, albeit a more complex one: https://github.com/traviscline/gevent-zeromq/blob/master/gevent_zeromq/__init__.py
Thank you,
Vitaly
How does gevent and tornado ioloop interact with each other? Will this setup be problematic?
Is there any way to limit maximum time taken to retrieve HTTP response when doing async requests through grequests.map? (no socket connection timeout itself). Monkey patching seems to break socket.setdefaulttimeout() and have played around with gevent.Timeout and similar but without really good results...
Thanks in advance
Maybe I'm overlooking a easy option here (just started using python again), but how would I fire off 100 requests with them all using the same session? Is it possible with the current API?
Ideally I'd like to be able to do something like
s = requests.session()
grequests.map(rs, session=s)
Anyone needed this before? How did you approach it?
Fully described here: http://stackoverflow.com/questions/11815640/memory-leak-with-grequests
Great lib!
But: in imap there is a size=2 as default, its very missleading. With size=2 parameter library works similar to blocking version. I would highly recommend changing it to maxint, if in map there is no throttling enabled.
The aggressive python stdlib monkey patching originating from https://github.com/kennethreitz/grequests/blob/master/grequests.py#L21 causes a lot of breakage throughout my app. The degree of monkey patching causes the redis client to error out which in turn then breaks all async celery processing.
I really like the grequests lib, but cannot use it if it's going to make things that used to work fine not work anymore.
File "/Users/jay/my-app/venv/lib/python2.7/site-packages/billiard/pool.py", line 1426, in safe_apply_callback
fun(*args)
File "/Users/jay/my-app/venv/lib/python2.7/site-packages/celery/worker/job.py", line 347, in on_success
self.acknowledge()
File "/Users/jay/my-app/venv/lib/python2.7/site-packages/celery/worker/job.py", line 453, in acknowledge
self.on_ack(logger, self.connection_errors)
File "/Users/jay/my-app/venv/lib/python2.7/site-packages/kombu/transport/base.py", line 100, in ack_log_error
self.ack()
File "/Users/jay/my-app/venv/lib/python2.7/site-packages/kombu/transport/base.py", line 95, in ack
self.channel.basic_ack(self.delivery_tag)
File "/Users/jay/my-app/venv/lib/python2.7/site-packages/kombu/transport/virtual/__init__.py", line 511, in basic_ack
self.qos.ack(delivery_tag)
File "/Users/jay/my-app/venv/lib/python2.7/site-packages/kombu/transport/redis.py", line 127, in ack
self._remove_from_indices(delivery_tag).execute()
File "/Users/jay/my-app/venv/lib/python2.7/site-packages/redis/client.py", line 1919, in execute
return execute(conn, stack, raise_on_error)
File "/Users/jay/my-app/venv/lib/python2.7/site-packages/redis/client.py", line 1811, in _execute_transaction
self.parse_response(connection, '_')
File "/Users/jay/my-app/venv/lib/python2.7/site-packages/redis/client.py", line 1882, in parse_response
self, connection, command_name, **options)
File "/Users/jay/my-app/venv/lib/python2.7/site-packages/redis/client.py", line 387, in parse_response
response = connection.read_response()
File "/Users/jay/my-app/venv/lib/python2.7/site-packages/redis/connection.py", line 307, in read_response
response = self._parser.read_response()
File "/Users/jay/my-app/venv/lib/python2.7/site-packages/redis/connection.py", line 105, in read_response
response = self.read()
File "/Users/jay/my-app/venv/lib/python2.7/site-packages/redis/connection.py", line 90, in read
return self._fp.readline()[:-2]
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 447, in readline
data = self._sock.recv(self._rbufsize)
File "/Users/jay/my-app/venv/lib/python2.7/site-packages/gevent/socket.py", line 392, in recv
self._wait(self._read_event)
File "/Users/jay/my-app/venv/lib/python2.7/site-packages/gevent/socket.py", line 298, in _wait
self.hub.wait(watcher)
File "/Users/jay/my-app/venv/lib/python2.7/site-packages/gevent/hub.py", line 341, in wait
result = waiter.get()
File "/Users/jay/my-app/venv/lib/python2.7/site-packages/gevent/hub.py", line 568, in get
return self.hub.switch()
File "/Users/jay/my-app/venv/lib/python2.7/site-packages/gevent/hub.py", line 331, in switch
return greenlet.switch(self)
LoopExit: This operation would block forever
Additional references pertaining to gevent monkey patching issues:
grequests breaks the multiprocessing.Manager() functionality. To replicate try to execute a script containing the following:
from multiprocessing import Manager
print Manager().Queue()
and then:
import grequests
from multiprocessing import Manager
print Manager().Queue()
:(
The following usage hangs grequests
:
>>> for response in grequests.imap(iter([])):
... print response
...
However, the same occurs with gevent
:
for x in Pool().imap_unordered(lambda x: x, iter([])):
... print x
...
So maybe this should be reported upstream?
When making a request with Grequests, a timeout error is not raised. Here is a failing test:
def test_concurrency_with_timeout(self):
n = 10
reqs = []
reqs.append(grequests.get(httpbin('delay/10'), timeout=1.5))
resp = grequests.map(reqs)
self.assertEqual([None], resp)
I am attempting to write a script that performs a bunch of requests asynchronously using grequest, and then spawns some threads that each use the standard python-requests module to perform some additional http requests. It seems that the first call to requests.get() after i've used grequest results in the following error:
NotImplementedError: gevent is only usable from a single thread
Is there something that needs to happen to use the python-requests module from a thread after using grequests?
import grequests as gr
gr.get('http://baidu.com')
Traceback (most recent call last):
File "", line 1, in
File "grequests.py", line 44, in wrapped
return f(_args, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests-1.0.4-py2.7.egg/requests/api.py", line 55, in get
return request('get', url, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests-1.0.4-py2.7.egg/requests/api.py", line 44, in request
return session.request(method=method, url=url, *_kwargs)
TypeError: request() got an unexpected keyword argument 'return_response'
I need to capture the request and an instance of an object that contains meta-data about the request to store in the database, how do you recommend that? You only return the response text.
Try to do a pip3 install grequests
with python 3.4.3, and am getting the following Traceback
Installing collected packages: gevent, grequests
Running setup.py install for gevent
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/private/var/folders/c_/nnqg051j4cb7f1_0jn06s1m00000gn/T/pip-build-r1efkdvz/gevent/setup.py", line 316, in <module>
run_setup(ext_modules)
File "/private/var/folders/c_/nnqg051j4cb7f1_0jn06s1m00000gn/T/pip-build-r1efkdvz/gevent/setup.py", line 311, in run_setup
"Development Status :: 4 - Beta"])
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/core.py", line 148, in setup
dist.run_commands()
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/dist.py", line 955, in run_commands
self.run_command(cmd)
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/usr/local/lib/python3.4/site-packages/setuptools-12.2-py3.4.egg/setuptools/command/install.py", line 61, in run
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/command/install.py", line 539, in run
self.run_command('build')
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/command/build.py", line 126, in run
self.run_command(cmd_name)
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/command/build_ext.py", line 339, in run
self.build_extensions()
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/command/build_ext.py", line 448, in build_extensions
self.build_extension(ext)
File "/private/var/folders/c_/nnqg051j4cb7f1_0jn06s1m00000gn/T/pip-build-r1efkdvz/gevent/setup.py", line 234, in build_extension
self.gevent_prepare(ext)
File "/private/var/folders/c_/nnqg051j4cb7f1_0jn06s1m00000gn/T/pip-build-r1efkdvz/gevent/setup.py", line 231, in gevent_prepare
configure(self, ext)
File "/private/var/folders/c_/nnqg051j4cb7f1_0jn06s1m00000gn/T/pip-build-r1efkdvz/gevent/setup.py", line 139, in configure_libev
make_universal_header('config.h', 'SIZEOF_LONG', 'SIZEOF_SIZE_T', 'SIZEOF_TIME_T')
File "/private/var/folders/c_/nnqg051j4cb7f1_0jn06s1m00000gn/T/pip-build-r1efkdvz/gevent/setup.py", line 111, in make_universal_header
print >>f, line
TypeError: unsupported operand type(s) for >>: 'builtin_function_or_method' and '_io.TextIOWrapper'
Running '/bin/sh /private/var/folders/c_/nnqg051j4cb7f1_0jn06s1m00000gn/T/pip-build-r1efkdvz/gevent/libev/configure > configure-output.txt' in /private/var/folders/c_/nnqg051j4cb7f1_0jn06s1m00000gn/T/pip-build-r1efkdvz/gevent/build/temp.macosx-10.10-x86_64-3.4/libev
Complete output from command /usr/local/opt/python3/bin/python3.4 -c "import setuptools, tokenize;__file__='/private/var/folders/c_/nnqg051j4cb7f1_0jn06s1m00000gn/T/pip-build-r1efkdvz/gevent/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/c_/nnqg051j4cb7f1_0jn06s1m00000gn/T/pip-2a9m9jp5-record/install-record.txt --single-version-externally-managed --compile:
running install
running build
running build_py
creating build
creating build/lib.macosx-10.10-x86_64-3.4
creating build/lib.macosx-10.10-x86_64-3.4/gevent
copying gevent/__init__.py -> build/lib.macosx-10.10-x86_64-3.4/gevent
copying gevent/_threading.py -> build/lib.macosx-10.10-x86_64-3.4/gevent
copying gevent/backdoor.py -> build/lib.macosx-10.10-x86_64-3.4/gevent
copying gevent/baseserver.py -> build/lib.macosx-10.10-x86_64-3.4/gevent
copying gevent/coros.py -> build/lib.macosx-10.10-x86_64-3.4/gevent
copying gevent/event.py -> build/lib.macosx-10.10-x86_64-3.4/gevent
copying gevent/fileobject.py -> build/lib.macosx-10.10-x86_64-3.4/gevent
copying gevent/greenlet.py -> build/lib.macosx-10.10-x86_64-3.4/gevent
copying gevent/hub.py -> build/lib.macosx-10.10-x86_64-3.4/gevent
copying gevent/local.py -> build/lib.macosx-10.10-x86_64-3.4/gevent
copying gevent/lock.py -> build/lib.macosx-10.10-x86_64-3.4/gevent
copying gevent/monkey.py -> build/lib.macosx-10.10-x86_64-3.4/gevent
copying gevent/os.py -> build/lib.macosx-10.10-x86_64-3.4/gevent
copying gevent/pool.py -> build/lib.macosx-10.10-x86_64-3.4/gevent
copying gevent/pywsgi.py -> build/lib.macosx-10.10-x86_64-3.4/gevent
copying gevent/queue.py -> build/lib.macosx-10.10-x86_64-3.4/gevent
copying gevent/resolver_ares.py -> build/lib.macosx-10.10-x86_64-3.4/gevent
copying gevent/resolver_thread.py -> build/lib.macosx-10.10-x86_64-3.4/gevent
copying gevent/select.py -> build/lib.macosx-10.10-x86_64-3.4/gevent
copying gevent/server.py -> build/lib.macosx-10.10-x86_64-3.4/gevent
copying gevent/socket.py -> build/lib.macosx-10.10-x86_64-3.4/gevent
copying gevent/ssl.py -> build/lib.macosx-10.10-x86_64-3.4/gevent
copying gevent/subprocess.py -> build/lib.macosx-10.10-x86_64-3.4/gevent
copying gevent/thread.py -> build/lib.macosx-10.10-x86_64-3.4/gevent
copying gevent/threading.py -> build/lib.macosx-10.10-x86_64-3.4/gevent
copying gevent/threadpool.py -> build/lib.macosx-10.10-x86_64-3.4/gevent
copying gevent/timeout.py -> build/lib.macosx-10.10-x86_64-3.4/gevent
copying gevent/util.py -> build/lib.macosx-10.10-x86_64-3.4/gevent
copying gevent/win32util.py -> build/lib.macosx-10.10-x86_64-3.4/gevent
copying gevent/wsgi.py -> build/lib.macosx-10.10-x86_64-3.4/gevent
running build_ext
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/private/var/folders/c_/nnqg051j4cb7f1_0jn06s1m00000gn/T/pip-build-r1efkdvz/gevent/setup.py", line 316, in <module>
run_setup(ext_modules)
File "/private/var/folders/c_/nnqg051j4cb7f1_0jn06s1m00000gn/T/pip-build-r1efkdvz/gevent/setup.py", line 311, in run_setup
"Development Status :: 4 - Beta"])
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/core.py", line 148, in setup
dist.run_commands()
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/dist.py", line 955, in run_commands
self.run_command(cmd)
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/usr/local/lib/python3.4/site-packages/setuptools-12.2-py3.4.egg/setuptools/command/install.py", line 61, in run
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/command/install.py", line 539, in run
self.run_command('build')
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/command/build.py", line 126, in run
self.run_command(cmd_name)
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/command/build_ext.py", line 339, in run
self.build_extensions()
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/command/build_ext.py", line 448, in build_extensions
self.build_extension(ext)
File "/private/var/folders/c_/nnqg051j4cb7f1_0jn06s1m00000gn/T/pip-build-r1efkdvz/gevent/setup.py", line 234, in build_extension
self.gevent_prepare(ext)
File "/private/var/folders/c_/nnqg051j4cb7f1_0jn06s1m00000gn/T/pip-build-r1efkdvz/gevent/setup.py", line 231, in gevent_prepare
configure(self, ext)
File "/private/var/folders/c_/nnqg051j4cb7f1_0jn06s1m00000gn/T/pip-build-r1efkdvz/gevent/setup.py", line 139, in configure_libev
make_universal_header('config.h', 'SIZEOF_LONG', 'SIZEOF_SIZE_T', 'SIZEOF_TIME_T')
File "/private/var/folders/c_/nnqg051j4cb7f1_0jn06s1m00000gn/T/pip-build-r1efkdvz/gevent/setup.py", line 111, in make_universal_header
print >>f, line
TypeError: unsupported operand type(s) for >>: 'builtin_function_or_method' and '_io.TextIOWrapper'
Running '/bin/sh /private/var/folders/c_/nnqg051j4cb7f1_0jn06s1m00000gn/T/pip-build-r1efkdvz/gevent/libev/configure > configure-output.txt' in /private/var/folders/c_/nnqg051j4cb7f1_0jn06s1m00000gn/T/pip-build-r1efkdvz/gevent/build/temp.macosx-10.10-x86_64-3.4/libev
----------------------------------------
Command "/usr/local/opt/python3/bin/python3.4 -c "import setuptools, tokenize;__file__='/private/var/folders/c_/nnqg051j4cb7f1_0jn06s1m00000gn/T/pip-build-r1efkdvz/gevent/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/c_/nnqg051j4cb7f1_0jn06s1m00000gn/T/pip-2a9m9jp5-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/var/folders/c_/nnqg051j4cb7f1_0jn06s1m00000gn/T/pip-build-r1efkdvz/gevent
Detailed description is in this SO question: https://stackoverflow.com/questions/23117176/too-many-urls-in-greqests-map-causes-gevent-hub-loopexit-only-on-mac , but I will answer any questions in comments to this issue if needed.
Just importing grequests
breaks requests.Session.post()
for me.
File "venv/lib/python2.7/site-packages/requests/sessions.py", line 508, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "venv/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
resp = self.send(prep, **send_kwargs)
File "venv/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "venv/lib/python2.7/site-packages/requests/adapters.py", line 370, in send
timeout=timeout
File "venv/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
body=body, headers=headers)
File "venv/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 341, in _make_request
self._validate_conn(conn)
File "venv/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 761, in _validate_conn
conn.connect()
File "venv/lib/python2.7/site-packages/requests/packages/urllib3/connection.py", line 238, in connect
ssl_version=resolved_ssl_version)
File "venv/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py", line 279, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 350, in wrap_socket
_context=self)
TypeError: __init__() got an unexpected keyword argument 'server_hostname'
Replicate:
url="http://www.google.com:81"# (chosen to cause no response)
item=grequests.get(url,hooks={'response':handle_result},timeout=20)
grequests.map(items, exception_handler=oops)
oops is not called but instead, eventually:
Traceback (most recent call last):
File "C:\Users\edwar_000\Desktop\quickloadtest.py", line 54, in
grequests.map(async_list, exception_handler=oops)
File "C:\Python27\lib\site-packages\grequests.py", line 123, in map
exception_handler(request, request.exception)
AttributeError: 'AsyncRequest' object has no attribute 'exception'
Before last commit, there was a way to put some bits of data inside kwargs['config'], that helped when retrieving request details from map() results. What is the preferred way to do that now?
The first call to grequests.map
takes five seconds longer than subsequent requests on my machine [1].
Example output from test case:
2012-10-15 16:32:03,769 starting...
2012-10-15 16:32:03,776 Starting new HTTP connection (1): www.bbc.co.uk
2012-10-15 16:32:03,776 Starting new HTTP connection (1): www.bbc.co.uk
2012-10-15 16:32:03,777 Starting new HTTP connection (1): www.bbc.co.uk
2012-10-15 16:32:08,829 "GET / HTTP/1.1" 200 24765
2012-10-15 16:32:08,841 "GET / HTTP/1.1" 200 24834
2012-10-15 16:32:08,850 "GET / HTTP/1.1" 200 25071
2012-10-15 16:32:08,885 took 5.116s
2012-10-15 16:32:08,889 starting...
2012-10-15 16:32:08,891 Starting new HTTP connection (1): www.bbc.co.uk
2012-10-15 16:32:08,892 Starting new HTTP connection (1): www.bbc.co.uk
2012-10-15 16:32:08,894 Starting new HTTP connection (1): www.bbc.co.uk
2012-10-15 16:32:08,968 "GET / HTTP/1.1" 200 25024
2012-10-15 16:32:08,973 "GET / HTTP/1.1" 200 25071
2012-10-15 16:32:08,977 "GET / HTTP/1.1" 200 24731
2012-10-15 16:32:09,005 took 0.116s
2012-10-15 16:32:09,009 starting...
2012-10-15 16:32:09,011 Starting new HTTP connection (1): www.bbc.co.uk
2012-10-15 16:32:09,012 Starting new HTTP connection (1): www.bbc.co.uk
2012-10-15 16:32:09,014 Starting new HTTP connection (1): www.bbc.co.uk
2012-10-15 16:32:09,085 "GET / HTTP/1.1" 200 24988
2012-10-15 16:32:09,089 "GET / HTTP/1.1" 200 24786
2012-10-15 16:32:09,090 "GET / HTTP/1.1" 200 24553
2012-10-15 16:32:09,121 took 0.112s
Test case:
import logging
import time
import grequests
logger = logging.getLogger(__name__)
logging.basicConfig(format='%(asctime)s %(message)s', level=logging.DEBUG)
urls = ['http://www.bbc.co.uk/'] * 3
for i in xrange(3):
requests = [grequests.get(url) for url in urls]
logger.info('starting...')
start = time.time()
responses = grequests.map(requests)
logger.info('took %.3fs' % (time.time() - start,))
[1] grequests version 0.1.0 installed via PyPi.
$ uname -a
Linux robson 2.6.35-32-generic-pae #67-Ubuntu SMP Mon Mar 5 21:23:19 UTC 2012 i686 GNU/Linux
$ python --version
Python 2.6.6
Hi, Is there anyway to turn the logging off? Errors like:
Timeout: (<requests.packages.urllib3.connectionpool.HTTPConnectionPool object at 0x1112bafd0>, 'Connection to 163.125.156.220 timed out. (connect timeout=10)')
<Greenlet at 0x11117d9b0: <bound method AsyncRequest.send of <grequests.AsyncRequest object at 0x1112e7210>>(stream=False)> failed with Timeout
are written in terminal. I found http://stackoverflow.com/questions/11029717/how-do-i-disable-log-messages-from-the-requests-library to silent the log messeges of requests, but it doesn't work for grequests. Any ideas? thanx.
See kennethreitz/requests#500 for context, but basically the timeout is no longer respected after importing the async module that monkeypatches socket
.
>>> import requests
>>> import time
>>> start = time.time(); r = requests.get('http://httpbin.org/delay/4', timeout=2.0); print time.time() - start, r.status_code
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "requests/api.py", line 52, in get
return request('get', url, **kwargs)
File "requests/api.py", line 40, in request
return s.request(method=method, url=url, **kwargs)
File "requests/sessions.py", line 229, in request
r.send(prefetch=prefetch)
File "requests/models.py", line 604, in send
raise Timeout('Request timed out.')
requests.exceptions.Timeout: Request timed out.
>>> start = time.time(); r = requests.get('http://httpbin.org/delay/4', timeout=2.0); print time.time() - start, r.status_code
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "requests/api.py", line 52, in get
return request('get', url, **kwargs)
File "requests/api.py", line 40, in request
return s.request(method=method, url=url, **kwargs)
File "requests/sessions.py", line 229, in request
r.send(prefetch=prefetch)
File "requests/models.py", line 604, in send
raise Timeout('Request timed out.')
requests.exceptions.Timeout: Request timed out.
>>> import requests.async
>>> start = time.time(); r = requests.get('http://httpbin.org/delay/4', timeout=2.0); print time.time() - start, r.status_code
4.38320612907 200
>>> start = time.time(); r = requests.get('http://httpbin.org/delay/4', timeout=2.0); print time.time() - start, r.status_code
4.67993688583 200
Basically the same issue as with requests
https://github.com/kennethreitz/requests/issues/239
When i tried to use exception handler i got this exception:
Process Process-1:
Traceback (most recent call last):
File "C:\Develop\Python27\lib\multiprocessing\process.py", line 258, in _bootstrap
self.run()
File "C:\Develop\Python27\lib\multiprocessing\process.py", line 114, in run
self._target(_self._args, *_self._kwargs)
File "C:\Users\TzAnAnY\GitProjects\AutoBrowser\nmap\nmap.py", line 149, in scan_progressive
callback(host, scan_data)
File "C:\Users\TzAnAnY\GitProjects\AutoBrowser\autoBrowsers.py", line 55, in callback_result
get_async_results('http', host, ports)
File "C:\Users\TzAnAnY\GitProjects\AutoBrowser\autoBrowsers.py", line 42, in get_async_results
exception_handler=exception_handler,
File "C:\Users\TzAnAnY\GitProjects\AutoBrowser\grequests.py", line 124, in map
exception_handler(request, request.exception)
AttributeError: 'AsyncRequest' object has no attribute 'exception'
Although a list of Requests is passed to the map
function, the actual HTTP requests are made in arbitrary order.
This is not unexpected, of course, but it would still be nice to have the list's order followed, somehow.
Running gevent.sleep(0)
after each spawn (i.e. in the send
function) seems to maintain order with no apparent side effects; perhaps you might like to consider adding it?
https://crate.io/packages/grequests/ isn't working
Hi Kenneth,
May I suggest that you consider using this instead of gevent:
http://docs.python.org/dev/library/concurrent.futures.html
It has a backport to previous versions
http://pypi.python.org/pypi/futures
It has a good interface IMO, it's in the standard library so will be supported indefinitely, it doesn't have external native dependencies like gevent does, and it can help avoid the problems stemming from gevent monkey patching as in issue 1. To be fair, gevent has awesome performance but it's just not as easy a dependency to work with as I'd like (I had to update to 1.0b2 to get pool.imap to work, but that version is not in pypi, etc.)
Hi there, we would like to update requests to 0.13.0, but we would need to know the exact licensing terms for grequest. 'BSD' isn't enough, as there are several versions (check http://spdx.org/licenses/). Also, the shipped LICENSE file doesn't seem to be BSD. Can you elaborate on that?
Grequest raise this exception "Gevent is required for grequests." when i use it with virtualenv(python 2.7.10 on linux ubuntu 14.04 LTS).
I was able to install grequests whitout having gevent. The dependency must be added to the setup.py so pip will try to install the dependency.
eg.
def map(requests, stream=False, size=None, exception_handler=None):
โฆโฆ
ret = []
for request in requests:
if request.response:
ret.append(request.response)
elif exception_handler:
#exception_handlerrequest, request.exception)
# new code:
ret.append(exception_handler(request))
return ret
And then I can use it like this:
from requests import Response
def timeOutException(request):
if 'ok' in dir(request.response):
return request.response
else:
# build respone for timeout
response = Response()
response.status_code = 444
response.url = request.url
return response
In [119]: res = grequests.map(apprs, exception_handler=timeOutException)
In [120]: res
Out[120]: [<Response [200]>, <Response [444]>, <Response [404]>, <Response [200]>]
The same process can also be in imap.
Mostly for convenience.
It looks like just gevent and request.
I wrote a test in which I use grequests to do a GET on my server 1000 times (throttled back to four simultaneous requests). The first 250 requests work fine, and then they stop returning data (and run very fast). Just before the run ends (after 995 of 1000 GETs), grequests raises this exception:
Traceback (most recent call last): File "submit_test.py", line 32, in File "submit_test.py", line 30, in main File "/Users/davidp/.virtualenvs/ssk_test/lib/python2.7/site-packages/grequests-0.1.0-py2.7.egg/grequests.py", line 83, in map File "/Users/davidp/.virtualenvs/ssk_test/lib/python2.7/site-packages/gevent-0.13.7-py2.7-macosx-10.6-intel.egg/gevent/greenlet.py", line 487, in joinall ImportError: No module named queue
I saw this same behavior with requests.async. It's not just my server because I get the same response when I use other sites.
My test was run with Python 2.7.3 on Mac OS X 10.7.4. grequests 0.1.0. gevent 0.13.7.
Here is my test code:
import time import grequests url = 'http://ssk-dev.appspot.com/test_service/default/search_unique' def process_page (content, account_id): global start if content: length = len(content) else: length = 0 print "In parse_page(): %d %d %f" % (account_id, length, time.time() - start) def main (): global start start = time.time() # Make a list of callback functions callback_list = [] for i in range (1000): def wrapper (id = i): # Fashion a closure because I need a bit of state def callback (response): process_page (response.content, id) return callback func = wrapper (i) callback_list.append (func) async_request_list = [grequests.get (url, hooks = {'response': func}) for func in callback_list] grequests.map (async_request_list, size = 4) main()
When I attempt to call map with 500 or more requests, usually a few of my requests timeout (I set it to 7 seconds).
This is ok but the error messages display and i'm not sure how to catch these errors because all the work is done individually in the map() method from grequests.
Can I either catch these timeouts individually somehow, or somehow log all the failed requests and re-attempt them? What is usually the protocol when sending out a bunch of requests asynchronously with the requirement that they all return back.
The returned response objects contain the redirected url, and I am unable to find the one that I requested for.
How do I find which returned object corresponds to which requested url?
In [31]: urls = [
'http://www.heroku.com',
'http://python-tablib.org',
'http://httpbin.org',
'http://python-requests.org',
'http://kennethreitz.com'
]
In [32]: rs = (grequests.get(u) for u in urls)
In [33]: for i in grequests.imap(rs):
print i, i.url
....:
<Response [200]> https://www.heroku.com/
<Response [200]> http://docs.python-tablib.org/en/latest/
<Response [200]> http://httpbin.org/
<Response [200]> http://docs.python-requests.org/en/latest/
<Response [200]> https://medium.com/@kennethreitz/collections/
neither response.url, nor response.request.url give the requested url.
In [35]: rs = (grequests.get(u) for u in urls)
In [36]: for i in grequests.imap(rs):
print i, i.request.url
....:
<Response [200]> https://www.heroku.com/
<Response [200]> http://docs.python-tablib.org/en/latest/
<Response [200]> http://httpbin.org/
<Response [200]> http://docs.python-requests.org/en/latest/
<Response [200]> https://medium.com/@kennethreitz/collections/
As can be seen if you try and run the example code from the README:
Python 2.6.5 (r265:79063, Oct 1 2012, 22:04:36)
Type "copyright", "credits" or "license" for more information.
IPython 0.10 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object'. ?object also works, ?? prints more.
In [1]: import requests
In [2]: requests.__version__
Out[2]: '1.0.3'
In [3]: import grequests
In [4]: urls = [
...: 'http://www.heroku.com',
...: 'http://tablib.org',
...: 'http://httpbin.org',
...: 'http://python-requests.org',
...: 'http://kennethreitz.com'
...: ]
In [5]: rs = (grequests.get(u) for u in urls)
In [6]: grequests.map(rs)
ERROR: An unexpected error occurred while tokenizing input
The following traceback may be corrupted or invalid
The error message is: ('EOF in multi-line statement', (3, 0))
ERROR: An unexpected error occurred while tokenizing input
The following traceback may be corrupted or invalid
The error message is: ('EOF in multi-line statement', (19, 0))
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
/root/requests_lib_upgrade/<ipython console> in <module>()
/usr/local/lib/python2.6/dist-packages/grequests.pyc in map(requests, prefetch, size)
77 """
78
---> 79 requests = list(requests)
80
81 pool = Pool(size) if size else None
/root/requests_lib_upgrade/<ipython console> in <genexpr>((u,))
/usr/local/lib/python2.6/dist-packages/grequests.pyc in wrapped(*args, **kwargs)
42 kwargs['config'] = config
43
---> 44 return f(*args, **kwargs)
45
46 return wrapped
/usr/local/lib/python2.6/dist-packages/requests/api.pyc in get(url, **kwargs)
47
48 kwargs.setdefault('allow_redirects', True)
---> 49 return request('get', url, **kwargs)
50
51
/usr/local/lib/python2.6/dist-packages/requests/api.pyc in request(method, url, **kwargs)
36
37 session = sessions.Session()
---> 38 return session.request(method=method, url=url, **kwargs)
39
40
TypeError: request() got an unexpected keyword argument 'return_response'
Unfortunately I cannot make grequests get installed in my 10.8 mac.
I have described my problem in stack overflow:
http://stackoverflow.com/questions/17865198/install-grequests-mac-os-10-8-gcc-error
Any possible workarounds?
i get this error TypeError:map() got an unexpected keyword argument 'exception_handler'
i am getting it on this line
manresponse =grequests.map(rs,exception_handler = eception_handler)
any help ??
I'm importing grequests (which then monkey patches networking libraries) in a multi-threaded Django app run under uWSGI and I get errors like this one:
Traceback (most recent call last):
File "/.../python2.7/site-packages/redis/connection.py", line 182, in read_response
buffer = self._sock.recv(4096)
File "/.../python2.7/site-packages/gevent/socket.py", line 392, in recv
self._wait(self._read_event)
File "/.../python2.7/site-packages/gevent/socket.py", line 298, in _wait
self.hub.wait(watcher)
File "/.../python2.7/site-packages/gevent/hub.py", line 341, in wait
result = waiter.get()
File "/.../python2.7/site-packages/gevent/hub.py", line 568, in get
return self.hub.switch()
File "/.../python2.7/site-packages/gevent/hub.py", line 331, in switch
return greenlet.switch(self)
LoopExit: This operation would block forever
Can this be caused by reading from sockets in multiple threads spawned by uWSGI or is there another explanation for this behaviour?
I'm using the map function to "unshorten" about 5.5 million t.co links. I give 2000 requests a time with a limit of 30. Afterwards I do the same for the unshortened links. Now once in a while t.co will send me back a whole lot of RST packets, each one for a different connection which had already been closed (FIN ACK and ACK) a few seconds earlier and usually my script is then requesting the unshortened links with the map function.
Usually this is not a problem, but sometimes the map function just stops doing anything afterwards and hangs the script.
First at all, thanks for porting requests to the asynchronous world, I loved it!
However I'm using Eventlet instead of Gevent so I adapted the code to make
it work with Eventlet. You can see it at https://github.com/jmg/erequests.
I was wondering if we can make a unified version where the library can
detect or be configured to use either Eventlet or Gevent based on the user
needs.
Many thanks.
I am using grequests to test the performance of my service. Both the service and the machine from which I start the requests are inside Amazon EC2. It seems that I cannot start more requests than ulimit -n
, even though I explicitly specify the size parameter for map() / imap()
My setup is roughly like this:
headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
params = [json.dumps({'url': urls[i]}) for i in urls]
reqs = [grequests.post(SERVICE_URL, data=params[i % len(params)], headers=headers)
for i in xrange(no_requests)]
res = grequests.imap(reqs, size=200)
for r in res:
...
r.close()
Doesn't the size
parameter here tell grequests
that it should issue no_requests, 200 at a time?
Can't find anything that solves my problem, I don't know what else to do.
Installing collected packages: gevent, grequests
Running setup.py install for gevent
Complete output from command /usr/bin/python -c "import setuptools, tokenize;__file__='/private/tmp/pip-build-09c1Gh/gevent/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-0hQM3Z-record/install-record.txt --single-version-externally-managed --compile:
running install
running build
running build_py
creating build
creating build/lib.macosx-10.10-intel-2.7
creating build/lib.macosx-10.10-intel-2.7/gevent
copying gevent/__init__.py -> build/lib.macosx-10.10-intel-2.7/gevent
copying gevent/_ssl2.py -> build/lib.macosx-10.10-intel-2.7/gevent
copying gevent/_sslgte279.py -> build/lib.macosx-10.10-intel-2.7/gevent
copying gevent/_threading.py -> build/lib.macosx-10.10-intel-2.7/gevent
copying gevent/backdoor.py -> build/lib.macosx-10.10-intel-2.7/gevent
copying gevent/baseserver.py -> build/lib.macosx-10.10-intel-2.7/gevent
copying gevent/coros.py -> build/lib.macosx-10.10-intel-2.7/gevent
copying gevent/event.py -> build/lib.macosx-10.10-intel-2.7/gevent
copying gevent/fileobject.py -> build/lib.macosx-10.10-intel-2.7/gevent
copying gevent/greenlet.py -> build/lib.macosx-10.10-intel-2.7/gevent
copying gevent/hub.py -> build/lib.macosx-10.10-intel-2.7/gevent
copying gevent/local.py -> build/lib.macosx-10.10-intel-2.7/gevent
copying gevent/lock.py -> build/lib.macosx-10.10-intel-2.7/gevent
copying gevent/monkey.py -> build/lib.macosx-10.10-intel-2.7/gevent
copying gevent/os.py -> build/lib.macosx-10.10-intel-2.7/gevent
copying gevent/pool.py -> build/lib.macosx-10.10-intel-2.7/gevent
copying gevent/pywsgi.py -> build/lib.macosx-10.10-intel-2.7/gevent
copying gevent/queue.py -> build/lib.macosx-10.10-intel-2.7/gevent
copying gevent/resolver_ares.py -> build/lib.macosx-10.10-intel-2.7/gevent
copying gevent/resolver_thread.py -> build/lib.macosx-10.10-intel-2.7/gevent
copying gevent/select.py -> build/lib.macosx-10.10-intel-2.7/gevent
copying gevent/server.py -> build/lib.macosx-10.10-intel-2.7/gevent
copying gevent/socket.py -> build/lib.macosx-10.10-intel-2.7/gevent
copying gevent/ssl.py -> build/lib.macosx-10.10-intel-2.7/gevent
copying gevent/subprocess.py -> build/lib.macosx-10.10-intel-2.7/gevent
copying gevent/thread.py -> build/lib.macosx-10.10-intel-2.7/gevent
copying gevent/threading.py -> build/lib.macosx-10.10-intel-2.7/gevent
copying gevent/threadpool.py -> build/lib.macosx-10.10-intel-2.7/gevent
copying gevent/timeout.py -> build/lib.macosx-10.10-intel-2.7/gevent
copying gevent/util.py -> build/lib.macosx-10.10-intel-2.7/gevent
copying gevent/win32util.py -> build/lib.macosx-10.10-intel-2.7/gevent
copying gevent/wsgi.py -> build/lib.macosx-10.10-intel-2.7/gevent
running build_ext
Running '/bin/sh /private/tmp/pip-build-09c1Gh/gevent/libev/configure > configure-output.txt' in /private/tmp/pip-build-09c1Gh/gevent/build/temp.macosx-10.10-intel-2.7/libev
building 'gevent.core' extension
creating build/temp.macosx-10.10-intel-2.7/gevent
cc -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -U__llvm__ -arch x86_64 -arch i386 -pipe -DLIBEV_EMBED=1 -DEV_COMMON= -DEV_CLEANUP_ENABLE=0 -DEV_EMBED_ENABLE=0 -DEV_PERIODIC_ENABLE=0 -Ibuild/temp.macosx-10.10-intel-2.7/libev -Ilibev -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c gevent/gevent.core.c -o build/temp.macosx-10.10-intel-2.7/gevent/gevent.core.o
In file included from gevent/gevent.core.c:249:
In file included from gevent/libev.h:2:
libev/ev.c:483:48: warning: '/*' within block comment [-Wcomment]
/*#define MIN_INTERVAL 0.00000095367431640625 /* 1/2**20, good till 2200 */
^
libev/ev.c:1029:42: error: '_Noreturn' keyword must precede function declarator
ecb_inline void ecb_unreachable (void) ecb_noreturn;
^~~~~~~~~~~~
_Noreturn
libev/ev.c:832:26: note: expanded from macro 'ecb_noreturn'
#define ecb_noreturn _Noreturn
^
libev/ev.c:1625:31: warning: 'extern' variable has an initializer [-Wextern-initializer]
EV_API_DECL struct ev_loop *ev_default_loop_ptr = 0; /* needs to be initialised to make it a definition despite extern */
^
libev/ev.c:1796:7: warning: unused variable 'ocur_' [-Wunused-variable]
array_needsize (ANPENDING, pendings [pri], pendingmax [pri], w_->pending, EMPTY2);
^
libev/ev.c:1758:22: note: expanded from macro 'array_needsize'
int ecb_unused ocur_ = (cur); \
^
libev/ev.c:1807:3: warning: unused variable 'ocur_' [-Wunused-variable]
array_needsize (W, rfeeds, rfeedmax, rfeedcnt + 1, EMPTY2);
^
libev/ev.c:1758:22: note: expanded from macro 'array_needsize'
int ecb_unused ocur_ = (cur); \
^
libev/ev.c:1934:7: warning: unused variable 'ocur_' [-Wunused-variable]
array_needsize (int, fdchanges, fdchangemax, fdchangecnt, EMPTY2);
^
libev/ev.c:1758:22: note: expanded from macro 'array_needsize'
int ecb_unused ocur_ = (cur); \
^
In file included from gevent/gevent.core.c:249:
In file included from gevent/libev.h:2:
In file included from libev/ev.c:2484:
libev/ev_kqueue.c:50:3: warning: unused variable 'ocur_' [-Wunused-variable]
array_needsize (struct kevent, kqueue_changes, kqueue_changemax, kqueue_changecnt, EMPTY2);
^
libev/ev.c:1758:22: note: expanded from macro 'array_needsize'
int ecb_unused ocur_ = (cur); \
^
In file included from gevent/gevent.core.c:249:
In file included from gevent/libev.h:2:
In file included from libev/ev.c:2490:
libev/ev_poll.c:66:7: warning: unused variable 'ocur_' [-Wunused-variable]
array_needsize (struct pollfd, polls, pollmax, pollcnt, EMPTY2);
^
libev/ev.c:1758:22: note: expanded from macro 'array_needsize'
int ecb_unused ocur_ = (cur); \
^
libev/ev.c:3648:34: warning: '&' within '|' [-Wbitwise-op-parentheses]
fd_change (EV_A_ fd, w->events & EV__IOFDSET | EV_ANFD_REIFY);
~~~~~~~~~~^~~~~~~~~~~~~ ~
libev/ev.c:3648:34: note: place parentheses around the '&' expression to silence this warning
fd_change (EV_A_ fd, w->events & EV__IOFDSET | EV_ANFD_REIFY);
^
( )
libev/ev.c:3687:3: warning: unused variable 'ocur_' [-Wunused-variable]
array_needsize (ANHE, timers, timermax, ev_active (w) + 1, EMPTY2);
^
libev/ev.c:1758:22: note: expanded from macro 'array_needsize'
int ecb_unused ocur_ = (cur); \
^
libev/ev.c:4367:5: warning: unused variable 'ocur_' [-Wunused-variable]
array_needsize (ev_idle *, idles [ABSPRI (w)], idlemax [ABSPRI (w)], active, EMPTY2);
^
libev/ev.c:1758:22: note: expanded from macro 'array_needsize'
int ecb_unused ocur_ = (cur); \
^
libev/ev.c:4407:3: warning: unused variable 'ocur_' [-Wunused-variable]
array_needsize (ev_prepare *, prepares, preparemax, preparecnt, EMPTY2);
^
libev/ev.c:1758:22: note: expanded from macro 'array_needsize'
int ecb_unused ocur_ = (cur); \
^
libev/ev.c:4445:3: warning: unused variable 'ocur_' [-Wunused-variable]
array_needsize (ev_check *, checks, checkmax, checkcnt, EMPTY2);
^
libev/ev.c:1758:22: note: expanded from macro 'array_needsize'
int ecb_unused ocur_ = (cur); \
^
libev/ev.c:4592:3: warning: unused variable 'ocur_' [-Wunused-variable]
array_needsize (ev_fork *, forks, forkmax, forkcnt, EMPTY2);
^
libev/ev.c:1758:22: note: expanded from macro 'array_needsize'
int ecb_unused ocur_ = (cur); \
^
libev/ev.c:4675:3: warning: unused variable 'ocur_' [-Wunused-variable]
array_needsize (ev_async *, asyncs, asyncmax, asynccnt, EMPTY2);
^
libev/ev.c:1758:22: note: expanded from macro 'array_needsize'
int ecb_unused ocur_ = (cur); \
^
14 warnings and 1 error generated.
error: command 'cc' failed with exit status 1
----------------------------------------
Command "/usr/bin/python -c "import setuptools, tokenize;__file__='/private/tmp/pip-build-09c1Gh/gevent/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-0hQM3Z-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/tmp/pip-build-09c1Gh/gevent```
As reported by @piotr-dobrogost at kennethreitz/requests#605 :
from requests import async
if __name__ == '__main__':
req = async.get('http://httpbin.org/get')
res = async.map([req])[0]
print res.content
produces (about 90% of the time):
Traceback (most recent call last):
File "C:\python\virtualenv\requests\lib\site-packages\gevent-0.13.7-py2.7-win-amd64.egg\gevent\greenlet.py", line 390, in run
result = self._run(*self.args, **self.kwargs)
File "C:\Users\Piotr\projects\requests\requests\models.py", line 606, in send
raise ConnectionError(e)
ConnectionError: HTTPConnectionPool(host='httpbin.org', port=80): Max retries exceeded with url: /get
<Greenlet at 0x35e59c8L: <bound method Request.send of <Request [GET]>>(prefetch=True)> failed with ConnectionError
None
in the following environment:
Requests 0.12.1
gevent 0.13.7-py2.7-win-amd64
greenlet 0.3.4-py2.7-win-amd64
Python 2.7.3 64bit
Windows Vista 64bit
Is it unwise to use both the grequests library and the requests library at the same time?
I ask as I've just seen an issue in my production environment. I have grequests installed for use in one section of my site, and use requests on a number of other pages. One of the simple requests.get() call has failed because it went down along the path of gevent and failed for some reason.
Interestingly, refreshing the page didn't produce this error and succeeded as it usually does.
[Wed Sep 03 00:32:58 2014] [error] File "/opt/tcg/cas/tcg-cas/local/lib/python2.7/site-packages/requests/api.py", line 55, in get
[Wed Sep 03 00:32:58 2014] [error] return request('get', url, **kwargs)
[Wed Sep 03 00:32:58 2014] [error] File "/opt/tcg/cas/tcg-cas/local/lib/python2.7/site-packages/requests/api.py", line 44, in request
[Wed Sep 03 00:32:58 2014] [error] return session.request(method=method, url=url, **kwargs)
[Wed Sep 03 00:32:58 2014] [error] File "/opt/tcg/cas/tcg-cas/local/lib/python2.7/site-packages/requests/sessions.py", line 456, in request
[Wed Sep 03 00:32:58 2014] [error] resp = self.send(prep, **send_kwargs)
[Wed Sep 03 00:32:58 2014] [error] File "/opt/tcg/cas/tcg-cas/local/lib/python2.7/site-packages/requests/sessions.py", line 559, in send
[Wed Sep 03 00:32:58 2014] [error] r = adapter.send(request, **kwargs)
[Wed Sep 03 00:32:58 2014] [error] File "/opt/tcg/cas/tcg-cas/local/lib/python2.7/site-packages/requests/adapters.py", line 327, in send
[Wed Sep 03 00:32:58 2014] [error] timeout=timeout
[Wed Sep 03 00:32:58 2014] [error] File "/opt/tcg/cas/tcg-cas/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 493, in urlopen
[Wed Sep 03 00:32:58 2014] [error] body=body, headers=headers)
[Wed Sep 03 00:32:58 2014] [error] File "/opt/tcg/cas/tcg-cas/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 291, in _make_request
[Wed Sep 03 00:32:58 2014] [error] conn.request(method, url, **httplib_request_kw)
[Wed Sep 03 00:32:58 2014] [error] File "/usr/lib/python2.7/httplib.py", line 958, in request
[Wed Sep 03 00:32:58 2014] [error] self._send_request(method, url, body, headers)
[Wed Sep 03 00:32:58 2014] [error] File "/usr/lib/python2.7/httplib.py", line 992, in _send_request
[Wed Sep 03 00:32:58 2014] [error] self.endheaders(body)
[Wed Sep 03 00:32:58 2014] [error] File "/usr/lib/python2.7/httplib.py", line 954, in endheaders
[Wed Sep 03 00:32:58 2014] [error] self._send_output(message_body)
[Wed Sep 03 00:32:58 2014] [error] File "/usr/lib/python2.7/httplib.py", line 814, in _send_output
[Wed Sep 03 00:32:58 2014] [error] self.send(msg)
[Wed Sep 03 00:32:58 2014] [error] File "/usr/lib/python2.7/httplib.py", line 776, in send
[Wed Sep 03 00:32:58 2014] [error] self.connect()
[Wed Sep 03 00:32:58 2014] [error] File "/opt/tcg/cas/tcg-cas/local/lib/python2.7/site-packages/requests/packages/urllib3/connection.py", line 106, in connect
[Wed Sep 03 00:32:58 2014] [error] conn = self._new_conn()
[Wed Sep 03 00:32:58 2014] [error] File "/opt/tcg/cas/tcg-cas/local/lib/python2.7/site-packages/requests/packages/urllib3/connection.py", line 90, in _new_conn
[Wed Sep 03 00:32:58 2014] [error] (self.host, self.port), self.timeout, *extra_args)
[Wed Sep 03 00:32:58 2014] [error] File "/opt/tcg/cas/tcg-cas/local/lib/python2.7/site-packages/gevent/socket.py", line 570, in create_connection
[Wed Sep 03 00:32:58 2014] [error] for res in getaddrinfo(host, port, 0 if has_ipv6 else AF_INET, SOCK_STREAM):
[Wed Sep 03 00:32:58 2014] [error] File "/opt/tcg/cas/tcg-cas/local/lib/python2.7/site-packages/gevent/socket.py", line 621, in getaddrinfo
[Wed Sep 03 00:32:58 2014] [error] return get_hub().resolver.getaddrinfo(host, port, family, socktype, proto, flags)
[Wed Sep 03 00:32:58 2014] [error] File "/opt/tcg/cas/tcg-cas/local/lib/python2.7/site-packages/gevent/resolver_thread.py", line 34, in getaddrinfo
[Wed Sep 03 00:32:58 2014] [error] return self.pool.apply_e(self.expected_errors, _socket.getaddrinfo, args, kwargs)
[Wed Sep 03 00:32:58 2014] [error] File "/opt/tcg/cas/tcg-cas/local/lib/python2.7/site-packages/gevent/threadpool.py", line 222, in apply_e
[Wed Sep 03 00:32:58 2014] [error] success, result = self.spawn(wrap_errors, expected_errors, function, args, kwargs).get()
[Wed Sep 03 00:32:58 2014] [error] File "/opt/tcg/cas/tcg-cas/local/lib/python2.7/site-packages/gevent/event.py", line 233, in get
[Wed Sep 03 00:32:58 2014] [error] result = self.hub.switch()
[Wed Sep 03 00:32:58 2014] [error] File "/opt/tcg/cas/tcg-cas/local/lib/python2.7/site-packages/gevent/hub.py", line 331, in switch
[Wed Sep 03 00:32:58 2014] [error] return greenlet.switch(self)
[Wed Sep 03 00:32:58 2014] [error] LoopExit: This operation would block forever
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.