theelous3 / asks Goto Github PK

View Code? Open in Web Editor NEW

508.0 12.0 63.0 885 KB

Async requests-like httplib for python.

License: MIT License

Python 100.00%

curio trio async async-requests requests http python3 concurrency network io

asks's People

Contributors

Stargazers

Watchers

Forkers

akuli raylu nickdirienzo yifeikong njsmith carlbordum miracle2k scottbelden dd-dent smurfix t-8ch joernheissler sztomi cmehay vincebackpack carlwgeorge agronholm jmfrank63 rnovatorov yijxiang l0x-c0d3z shellsec nonsleepr clamor-py bottom-lover gridl cxhuan gwerbin samuelirvine purplemyst millefalcon dahlia rshatch jean poqoid dabeaz fas96 asl2 ahmedhaies wuqd2048 lagerdata matham apexo rmstorm palkeo vcalvert gmacon devsoftbank cybernetics abloch emtrane p-unity-lineage yxlwfds cauebs rvencu neverlandtf qk21126215 ahumeau pyxvlad djun tarlano arpitjain799

asks's Issues

Behavior of timeout when streaming

Hello there,

timeout seems to apply only on connection but not on received data when stream is enabled.

Here an example:

server.py

import time
from flask import Flask, Response

app = Flask(__name__)


@app.route('/')
def ping():
    def generate():
        while True:
            yield "PING\n"
            time.sleep(10)
    return Response(generate(), mimetype='text/plain')

run this

$ FLASK_APP=server.py flask run

Server will echo PING every 10 seconds

asks_test.py

import asks
import trio

asks.init('trio')


async def test_timeout():
    r = await asks.get(
        'http://localhost:5000/',
        stream=True,
        timeout=5,
        retries=0,
    )

    async for line in r.body:
        print(line)


trio.run(test_timeout)

The connection should timeout after 5 seconds, but it doesn't.

$ python asks_stream_timeout.py
bytearray(b'PING\n')
bytearray(b'PING\n')
bytearray(b'PING\n')

There is no way to handle a connection lost in this case.

I'm starting learning using async development with python, but I don't find a workaround using trio to handle a timeout if the connection has been lost.

Where's the logging at?

It'd sure be handy if logging was used throughout asks for debugging purposes 👍

bug with timeout manager

ERROR:curio.debug:Task 5209 crashed
Traceback (most recent call last):
  File "/home/kongyifei/repos/crawlet/.venv/lib/python3.6/site-packages/h11/_state.py", line 249, in _fire_event_triggered_transitions
    new_state = EVENT_TRIGGERED_TRANSITIONS[role][state][event_type]
KeyError: <class 'h11._events.ConnectionClosed'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/kongyifei/repos/crawlet/.venv/src/asks/asks/request.py", line 524, in _catch_response
    response = await self._recv_event(hconnection)
  File "/home/kongyifei/repos/crawlet/.venv/src/asks/asks/request.py", line 583, in _recv_event
    event = hconnection.next_event()
  File "/home/kongyifei/repos/crawlet/.venv/lib/python3.6/site-packages/h11/_connection.py", line 439, in next_event
    exc._reraise_as_remote_protocol_error()
  File "/home/kongyifei/repos/crawlet/.venv/lib/python3.6/site-packages/h11/_util.py", line 71, in _reraise_as_remote_protocol_error
    raise self
  File "/home/kongyifei/repos/crawlet/.venv/lib/python3.6/site-packages/h11/_connection.py", line 422, in next_event
    self._process_event(self.their_role, event)
  File "/home/kongyifei/repos/crawlet/.venv/lib/python3.6/site-packages/h11/_connection.py", line 238, in _process_event
    self._cstate.process_event(role, type(event), server_switch_event)
  File "/home/kongyifei/repos/crawlet/.venv/lib/python3.6/site-packages/h11/_state.py", line 238, in process_event
    self._fire_event_triggered_transitions(role, event_type)
  File "/home/kongyifei/repos/crawlet/.venv/lib/python3.6/site-packages/h11/_state.py", line 253, in _fire_event_triggered_transitions
    .format(event_type, role, self.states[role]))
h11._util.RemoteProtocolError: can't handle event type <class 'h11._events.ConnectionClosed'> for SERVER in state SEND_RESPONSE

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/kongyifei/repos/crawlet/.venv/lib/python3.6/site-packages/curio/kernel.py", line 826, in _run_coro
    trap = current._send(current.next_value)
  File "/home/kongyifei/repos/crawlet/.venv/lib/python3.6/site-packages/curio/task.py", line 96, in _task_runner
    return await coro
  File "/home/kongyifei/repos/crawlet/.venv/src/asks/asks/request.py", line 203, in make_request
    response_obj = await self._request_io(req, req_body, hconnection)
  File "/home/kongyifei/repos/crawlet/.venv/src/asks/asks/request.py", line 240, in _request_io
    response_obj = await self._catch_response(hconnection)
  File "/home/kongyifei/repos/crawlet/.venv/src/asks/asks/request.py", line 527, in _catch_response
    response = await self._recv_event(hconnection)
  File "/home/kongyifei/repos/crawlet/.venv/src/asks/asks/request.py", line 583, in _recv_event
    event = hconnection.next_event()
  File "/home/kongyifei/repos/crawlet/.venv/lib/python3.6/site-packages/h11/_connection.py", line 418, in next_event
    "Can't receive data when peer state is ERROR")
h11._util.RemoteProtocolError: Can't receive data when peer state is ERROR
ERROR:Crawlet-spider:download error for https://bj.lianjia.com/ershoufang/101101673238.html, TaskGroupError(RemoteProtocolError)
Traceback (most recent call last):
  File "/home/kongyifei/repos/crawlet/crawlet/download.py", line 71, in process_request
    timeout=context.rule.get('timeout', 10),
  File "/home/kongyifei/repos/crawlet/.venv/src/asks/asks/sessions.py", line 158, in request
    sock, r = await self.timeout_manager(timeout, req_obj)
  File "/home/kongyifei/repos/crawlet/.venv/src/asks/asks/sessions.py", line 187, in timeout_manager
    req_obj.make_request)
  File "/home/kongyifei/repos/crawlet/.venv/lib/python3.6/site-packages/curio/task.py", line 446, in __aexit__
    await self.join(wait=self._wait)
  File "/home/kongyifei/repos/crawlet/.venv/lib/python3.6/site-packages/curio/task.py", line 435, in join
    raise TaskGroupError(exceptional)
curio.errors.TaskGroupError: TaskGroupError(RemoteProtocolError)

Weird behavior with Session.get and streaming?

I might be doing something wrong, but I can't figure this out.

I have the following code that talks to a pypi simple API (think self.session.base_location being set to https://pypi.org/simple/):

  async def get(self, path, stream=False):
    self.log.debug(f"GET {path}")
    self.log.debug(f"stream = {stream}")
    return await self.session.get(path=path, auth=self.auth, stream=stream)

Now, I'm doing this:

await api.get("/requests")   # (1)

I just added the stream parameter with the default value of False. And suddenly my code expecting a Response breaks because the return value is now StreamResponse (despite stream being False). My log output:

2018-09-22 13:41:51,433   bib.installer.installer  DEBUG    - Base URL: https://pypi.org/simple
2018-09-22 13:41:51,433   bib.installer.installer  DEBUG    - Getting metadata of requests
2018-09-22 13:41:51,433   bib.api.legacy           DEBUG    - GET /requests
2018-09-22 13:41:51,433   bib.api.legacy           DEBUG    - stream = False
Traceback (most recent call last):
  File "/home/tamas/.local/share/virtualenvs/bib-wgx9oSkl/bin/bib", line 11, in <module>
    load_entry_point('bib', 'console_scripts', 'bib')()
  File "/home/tamas/.local/share/virtualenvs/bib-wgx9oSkl/lib/python3.7/site-packages/trio_click/core.py", line 748, in __call__
    return trio.run(main,*args)
  File "/home/tamas/.local/share/virtualenvs/bib-wgx9oSkl/lib/python3.7/site-packages/trio/_core/_run.py", line 1328, in run
    raise runner.main_task_outcome.error
  File "/home/tamas/.local/share/virtualenvs/bib-wgx9oSkl/lib/python3.7/site-packages/trio_click/core.py", line 720, in main
    rv = await self.invoke(ctx)
  File "/home/tamas/.local/share/virtualenvs/bib-wgx9oSkl/lib/python3.7/site-packages/trio_click/core.py", line 1117, in invoke
    return await _process_result(await sub_ctx.command.invoke(sub_ctx))
  File "/home/tamas/.local/share/virtualenvs/bib-wgx9oSkl/lib/python3.7/site-packages/trio_click/core.py", line 936, in invoke
    return await ctx.invoke(self.callback, **ctx.params)
  File "/home/tamas/.local/share/virtualenvs/bib-wgx9oSkl/lib/python3.7/site-packages/trio_click/core.py", line 560, in invoke
    rv = await rv
  File "/home/tamas/projects/bib/bib/cli.py", line 19, in install
    await installer.fetch_pkg(pkg)
  File "/home/tamas/projects/bib/bib/installer/installer.py", line 65, in fetch_pkg
    url = await self._fetch_metadata(pkg)
  File "/home/tamas/projects/bib/bib/installer/installer.py", line 37, in _fetch_metadata
    releases = await self.api.get_releases(pkg)
  File "/home/tamas/projects/bib/bib/api/legacy.py", line 22, in get_releases
    data = await trio.run_sync_in_worker_thread(parse_simple_index, resp.text)
AttributeError: 'StreamResponse' object has no attribute 'text'

However, if I do this:

  async def get(self, path, stream=False):
    self.log.debug(f"GET {path}")
    self.log.debug(f"stream = {stream}")
    if stream:
      return await self.session.get(path=path, auth=self.auth, stream=stream)
    return await self.session.get(path=path, auth=self.auth)

i.e. not pass stream if it's false, the line marked with (1) works as expected and this error goes away.

Unlike the code which previously used asks.get for a streaming response. I changed it to use the above function (to reuse the session). Now, even though I pass stream=True in this case, I still get Response/bytes instead of StreamResponse:

whl_resp = await self.api.get(url, stream=True)  # (2)
fname = url.split("/")[-1]
output_name = trio.Path(self.target_dir / fname)
self.log.debug(type(whl_resp))
self.log.debug(type(whl_resp.body))

My log output for this second code block:

2018-09-22 13:34:53,307   bib.api.legacy           DEBUG    - GET /requests-2.19.1-py2.py3-none-any.whl
2018-09-22 13:34:53,307   bib.api.legacy           DEBUG    - stream = True
2018-09-22 13:34:53,592   bib.installer.installer  DEBUG    - <class 'asks.response_objects.Response'>
2018-09-22 13:34:53,592   bib.installer.installer  DEBUG    - <class 'bytes'>

As a result, my code that follows this example for streamed downloads breaks (because the body is bytes).

I would expect the following behavior:

If I pass stream=True to Session.get, the response becomes a StreamResponse (regardless of actual streaming taking place)
If I pass stream=False or don't pass stream at all, the response is Response and body is bytes

Am I doing something wrong or is this a bug?

If there is a port number, it gives an error

`
import asks
import curio
asks.init('curio')

async def example(url):
r = await asks.get(url)
print(r.content[:20])
return r

curio.run (example('http://176.9.8.16:8081'))
`

Fix trio example in Readme

Fix so that the example runs (right syntax for start_soon) and demonstrates parallel requests (more than 1 connection for Session):

import asks
import trio
asks.init('trio')

path_list = ['a', 'list', 'of', '1000', 'paths']

results = []

async def grabber(path):
    r = await s.get(path)
    results.append(r)

async def main(path_list):
    async with trio.open_nursery() as n:
        for path in path_list:
            n.start_soon(grabber, path)

s = asks.Session(connections=100)
trio.run(main, path_list)

Cyrillic error

import asks
import curio

async def grabber(path, s):
print ('grabber',path, s)
r = await s.get(path=path)
print (r.content)

async def main():
path_list = ['/?cyrillic=ЛОРПЛОПР']
#path_list = ['/?non_cyrillic=HGJHGJHG']
s = asks.HSession(host='https://example.org')
for path in path_list:
await curio.spawn(grabber,path, s)

curio.run(main())

...
File "/home/wku/anaconda3/lib/python3.6/site-packages/h11/_util.py", line 119, in bytesify
s = s.encode("ascii")
UnicodeEncodeError: 'ascii' codec can't encode characters in position 11-18: ordinal not in range(128)

h11 They also wrote about the error

Problem with socket, OSError: [Errno 107] Transport endpoint is not connected

import asks
import curio

async def example():
r = await asks.get('https://ya.ru')#does not work
# r = await asks.get('https://yandex.ru')#does not work
# r = await asks.get('https://auto.ru')#does not work

# r = await asks.get('https://www.bing.com/')#works correctly
# r = await asks.get('https://google.com')#works correctly
print(r.content)

curio.run(example())

wku@wku:~/test$ python main.py
Task 2 crashed
Traceback (most recent call last):
File "/home/wku/anaconda3/lib/python3.6/site-packages/curio/kernel.py", line 826, in _run_coro
trap = current._send(current.next_value)
File "/home/wku/anaconda3/lib/python3.6/site-packages/curio/task.py", line 96, in _task_runner
return await coro
File "main.py", line 5, in example
r = await asks.get('https://ya.ru')
File "/home/wku/anaconda3/lib/python3.6/site-packages/asks/base_funcs.py", line 29, in request
r = await s.request(method, url=uri, **kwargs)
File "/home/wku/anaconda3/lib/python3.6/site-packages/asks/sessions.py", line 139, in request
sock = await self._grab_connection(url)
File "/home/wku/anaconda3/lib/python3.6/site-packages/asks/sessions.py", line 382, in _grab_connection
sock = await self._make_connection(host_loc)
File "/home/wku/anaconda3/lib/python3.6/site-packages/asks/sessions.py", line 355, in _make_connection
sock, port = await self._connect(host_loc=host_loc)
File "/home/wku/anaconda3/lib/python3.6/site-packages/asks/sessions.py", line 70, in _connect
(netloc, int(port))), port
File "/home/wku/anaconda3/lib/python3.6/site-packages/asks/sessions.py", line 49, in _open_connection_https
server_hostname=location[0])
File "/home/wku/anaconda3/lib/python3.6/site-packages/curio/network.py", line 74, in open_connection
sock = await _wrap_ssl_client(sock, ssl, server_hostname, alpn_protocols)
File "/home/wku/anaconda3/lib/python3.6/site-packages/curio/network.py", line 58, in _wrap_ssl_client
await sock.do_handshake()
File "/home/wku/anaconda3/lib/python3.6/site-packages/curio/io.py", line 282, in do_handshake
return self._socket.do_handshake()
File "/home/wku/anaconda3/lib/python3.6/ssl.py", line 1056, in do_handshake
self._check_connected()
File "/home/wku/anaconda3/lib/python3.6/ssl.py", line 855, in _check_connected
self.getpeername()
OSError: [Errno 107] Transport endpoint is not connected
Traceback (most recent call last):
File "/home/wku/anaconda3/lib/python3.6/site-packages/curio/kernel.py", line 826, in _run_coro
trap = current._send(current.next_value)
File "/home/wku/anaconda3/lib/python3.6/site-packages/curio/task.py", line 96, in _task_runner
return await coro
File "main.py", line 5, in example
r = await asks.get('https://ya.ru')
File "/home/wku/anaconda3/lib/python3.6/site-packages/asks/base_funcs.py", line 29, in request
r = await s.request(method, url=uri, **kwargs)
File "/home/wku/anaconda3/lib/python3.6/site-packages/asks/sessions.py", line 139, in request
sock = await self._grab_connection(url)
File "/home/wku/anaconda3/lib/python3.6/site-packages/asks/sessions.py", line 382, in _grab_connection
sock = await self._make_connection(host_loc)
File "/home/wku/anaconda3/lib/python3.6/site-packages/asks/sessions.py", line 355, in _make_connection
sock, port = await self._connect(host_loc=host_loc)
File "/home/wku/anaconda3/lib/python3.6/site-packages/asks/sessions.py", line 70, in _connect
(netloc, int(port))), port
File "/home/wku/anaconda3/lib/python3.6/site-packages/asks/sessions.py", line 49, in _open_connection_https
server_hostname=location[0])
File "/home/wku/anaconda3/lib/python3.6/site-packages/curio/network.py", line 74, in open_connection
sock = await _wrap_ssl_client(sock, ssl, server_hostname, alpn_protocols)
File "/home/wku/anaconda3/lib/python3.6/site-packages/curio/network.py", line 58, in _wrap_ssl_client
await sock.do_handshake()
File "/home/wku/anaconda3/lib/python3.6/site-packages/curio/io.py", line 282, in do_handshake
return self._socket.do_handshake()
File "/home/wku/anaconda3/lib/python3.6/ssl.py", line 1056, in do_handshake
self._check_connected()
File "/home/wku/anaconda3/lib/python3.6/ssl.py", line 855, in _check_connected
self.getpeername()
OSError: [Errno 107] Transport endpoint is not connected

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "main.py", line 8, in
curio.run(example())
File "/home/wku/anaconda3/lib/python3.6/site-packages/curio/kernel.py", line 877, in run
return kernel.run(corofunc, *args, timeout=timeout)
File "/home/wku/anaconda3/lib/python3.6/site-packages/curio/kernel.py", line 214, in run
raise TaskError('Task Crashed') from ret_exc
curio.errors.TaskError: Task Crashed
wku@wku:~/test$

Possible bug in Session._checked_out_sockets

Working on an async API downloader for work, my current crack at it uses asks and trio.

Randomly, the following statement:

response = await self._session.get(params=payload, timeout=300)

could inspire this:

...(Loads of random stuff...)
File "~\code\shopstyle-plugin\src\shopstyle_plugin\downloader.py", line 110, in _safe_fetch
    response = await self._session.get(params=payload, timeout=3000)
File "~\code\shopstyle-plugin\venv\lib\site-packages\asks\sessions.py", line 171, in request
    await self._replace_connection(sock)
File "~\code\shopstyle-plugin\venv\lib\site-packages\asks\sessions.py", line 255, in _replace_connection
    self._checked_out_sockets.remove(sock)
ValueError: deque.remove(x): x not in deque

I only went as far out to see what deque was problematic, which is the field mentioned in the title, which in turn is an instance of SocketQ(deque).
So... am I looking at a bug or misuse here?

convenience attribute `Response.ok`

requests Response objects have an ok attribute that I enjoy using.

http://docs.python-requests.org/en/master/api/#requests.Response.ok

Please implement this in asks.

Error when working with cookies

import asks
import curio
asks.init('curio')
from asks.sessions import Session

async def hsession_t_stateful(s):
r = await s.get()
assert r.status_code == 200

async def main():
    s = Session('https://google.ie', persist_cookies=True)
    async with curio.TaskGroup() as g:
        for _ in range(2):
            await curio.spawn(hsession_t_stateful(s))

curio.run(main())

"""
...
File "/home/wku/anaconda3/lib/python3.6/site-packages/asks/cookie_utils.py", line 34, in _check_cookies
return self._get_cookies_to_send(relevant_domains)
File "/home/wku/anaconda3/lib/python3.6/site-packages/asks/cookie_utils.py", line 46, in _get_cookies_to_send
cookies_to_go[cookie_obj.name] = cookie_obj.value
AttributeError: 'str' object has no attribute 'value'
...

"""

Please fix `pydoc3 asks.sessions`

$ pydoc3 asks.sessions
Traceback (most recent call last):
  File "/usr/bin/pydoc3", line 5, in <module>
    pydoc.cli()
  File "/usr/lib/python3.5/pydoc.py", line 2623, in cli
    help.help(arg)
  File "/usr/lib/python3.5/pydoc.py", line 1903, in help
    elif request: doc(request, 'Help on %s:', output=self._output)
  File "/usr/lib/python3.5/pydoc.py", line 1640, in doc
    pager(render_doc(thing, title, forceload))
  File "/usr/lib/python3.5/pydoc.py", line 1633, in render_doc
    return title % desc + '\n\n' + renderer.document(object, name)
  File "/usr/lib/python3.5/pydoc.py", line 374, in document
    if inspect.ismodule(object): return self.docmodule(*args)
  File "/usr/lib/python3.5/pydoc.py", line 1113, in docmodule
    for key, value in inspect.getmembers(object, inspect.isclass):
  File "/usr/lib/python3.5/inspect.py", line 316, in getmembers
    if not predicate or predicate(value):
  File "/usr/lib/python3.5/inspect.py", line 78, in isclass
    return isinstance(object, type)
  File "/usr/lib/python3/dist-packages/multio/__init__.py", line 397, in __getattribute__
    raise RuntimeError("multio.init() wasn't called")
RuntimeError: multio.init() wasn't called

HEY USERS :D UPDATE YOUR MULTIO BEFORE POSTING .CLOSE() ISSUES :D

asks is not decoding the response when streaming

It seems it is currently not supported to have the library decode a streaming gzip response.

Make Response objects usable as async context managers

It would be nice if the Response object would implement the async context manager protocol, even as a NOOP.
When migrating from aiohttp/requests code often treats responses as such and this would reduce the migration friction to asks.
Furthermore it would futureproof usercode in case future, more sophisticated Response object actually need resource managers.

asks 1.3.9 + Trio 0.3.0 + stream seems to be broken?

$ pip freeze | grep -e trio -e curio -e asks
asks==1.3.9
curio==0.8
trio==0.3.0

Example:

import io
import asks
import trio
import curio

async def download_file(url, fd):
    """ Download file and save it to the provided fd. """
    r = await asks.get(url, stream=True)
    async with r.body:
        async for bytechunk in r.body:
            fd.write(bytechunk)
    fd.seek(0)

url = 'https://www.sec.gov/Archives/edgar/full-index/1993/QTR1/master.gz'

asks.init('curio')
fd = io.BytesIO()
curio.run(download_file, url, fd)
no_bytes = len(fd.read())
assert no_bytes == 446, no_bytes
print("curio OK")

asks.init('trio')
fd = io.BytesIO()
trio.run(download_file, url, fd)
no_bytes = len(file.read())
assert no_bytes == 446, no_bytes
print("trio OK")

The curio code downloads the file just fine, but trio throws an exception.

Traceback (most recent call last):
  File "tri.py", line 58, in <module>
    trio.run(download_file, url, fd)
  File "/home/feanor/.virtualenvs/xbrl/lib/python3.6/site-packages/trio/_core/_run.py", line 1225, in run
    return result.unwrap()
  File "/home/feanor/.virtualenvs/xbrl/lib/python3.6/site-packages/trio/_core/_result.py", line 119, in unwrap
    raise self.error
  File "/home/feanor/.virtualenvs/xbrl/lib/python3.6/site-packages/trio/_core/_run.py", line 1334, in run_impl
    msg = task.coro.send(next_send)
  File "/home/feanor/.virtualenvs/xbrl/lib/python3.6/site-packages/trio/_core/_run.py", line 923, in init
    self.entry_queue.spawn()
  File "/home/feanor/.virtualenvs/xbrl/lib/python3.6/site-packages/trio/_util.py", line 109, in __aexit__
    await self._agen.asend(None)
  File "/home/feanor/.virtualenvs/xbrl/lib/python3.6/site-packages/async_generator/_impl.py", line 274, in asend
    return await self._do_it(self._it.send, value)
  File "/home/feanor/.virtualenvs/xbrl/lib/python3.6/site-packages/async_generator/_impl.py", line 290, in _do_it
    return await ANextIter(self._it, start_fn, *args)
  File "/home/feanor/.virtualenvs/xbrl/lib/python3.6/site-packages/async_generator/_impl.py", line 202, in send
    return self._invoke(self._it.send, value)
  File "/home/feanor/.virtualenvs/xbrl/lib/python3.6/site-packages/async_generator/_impl.py", line 209, in _invoke
    result = fn(*args)
  File "/home/feanor/.virtualenvs/xbrl/lib/python3.6/site-packages/trio/_core/_run.py", line 318, in open_nursery
    await nursery._nested_child_finished(nested_child_exc)
  File "/usr/lib64/python3.6/contextlib.py", line 99, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/feanor/.virtualenvs/xbrl/lib/python3.6/site-packages/trio/_core/_run.py", line 203, in open_cancel_scope
    yield scope
  File "/home/feanor/.virtualenvs/xbrl/lib/python3.6/site-packages/trio/_core/_multierror.py", line 144, in __exit__
    raise filtered_exc
  File "/home/feanor/.virtualenvs/xbrl/lib/python3.6/site-packages/trio/_core/_run.py", line 203, in open_cancel_scope
    yield scope
  File "/home/feanor/.virtualenvs/xbrl/lib/python3.6/site-packages/trio/_core/_run.py", line 318, in open_nursery
    await nursery._nested_child_finished(nested_child_exc)
  File "/home/feanor/.virtualenvs/xbrl/lib/python3.6/site-packages/trio/_core/_run.py", line 427, in _nested_child_finished
    raise MultiError(self._pending_excs)
  File "/home/feanor/.virtualenvs/xbrl/lib/python3.6/site-packages/trio/_core/_run.py", line 1334, in run_impl
    msg = task.coro.send(next_send)
  File "tri.py", line 43, in download_file
    fd.write(bytechunk)
  File "/home/feanor/.virtualenvs/xbrl/lib/python3.6/site-packages/asks/response_objects.py", line 186, in __aexit__
    await self.close()
  File "/home/feanor/.virtualenvs/xbrl/lib/python3.6/site-packages/asks/response_objects.py", line 183, in close
    await self.sock.close()
  File "/home/feanor/.virtualenvs/xbrl/lib/python3.6/site-packages/trio/_ssl.py", line 399, in __getattr__
    raise AttributeError(name)
AttributeError: close

Slow tests

Tests are phenomenally slow. Actual performance seems fine.

Investigate!

┌─╼[~/zoo/python/asks/tests] [master*] 
└────╼ pytest --tb=short
============================================== test session starts ==============================================
platform linux -- Python 3.6.3, pytest-3.2.3, py-1.4.34, pluggy-0.4.0
rootdir: /home/tl3/zoo/python/asks, inifile:
plugins: httpbin-0.3.0
collected 56 items                                                                                               

test_asks_curio.py ..........................
test_asks_trio.py ..........................
test_request_object.py ..
test_response_objects.py ..

========================================== 56 passed in 117.17 seconds ==========================================

Can asks add HOST parameters automatically?

I connect to chromium's debug port
resp = await asks.get("http://127.0.0.1:9222")
print(resp.text)
I see result is:
'Host header is specified and is not an IP address or localhost.'
then I modity code to:
resp = await asks.get("http://127.0.0.1:9222", headers={"HOST":"127.0.0.1:9222"})
print(resp)
I see correct result:
<Response 200 OK>

Can asks add HOST parameters automatically?
I using requests.get, no problem!

Add trio support

I know it is already on the radar but thought I would put it in as an issue. It appears trio now has (unreleased) async file io :-)

update documentation to include retries arg

Switch out cookie handling in favor of stdlib `http.cookies`

Mentioned in #22 (comment).

Decided to create a separate ticket for tracking this refactor. Still wrapping my head around the codebase, but I think @theelous3 should be able to explain a little more about what's going on here.

I'm not even going to try to guess right now given how much of a noob I am around here, ha.

Error: Embedded null byte in file when uploading bytes

Traceback (most recent call last):
  File "/home/laura/.local/share/virtualenvs/jokusoramame-SR0dS4Ee/lib/python3.6/site-packages/curious/commands/context.py", line 265, in invoke
    return await matched_command(self, *converted_args, **converted_kwargs)
  File "/home/laura/dev/discord/jokusoramame/jokusoramame/plugins/analytics.py", line 282, in command_server_distribution
    await ctx.channel.messages.upload(Path("/tmp/plot.png"), filename="plot.png")
  File "/home/laura/.local/share/virtualenvs/jokusoramame-SR0dS4Ee/lib/python3.6/site-packages/curious/dataclasses/channel.py", line 351, in upload
    filename=filename, content=message_content)
  File "/home/laura/.local/share/virtualenvs/jokusoramame-SR0dS4Ee/lib/python3.6/site-packages/curious/core/httpclient.py", line 614, in send_file
    body=payload, files=files)
  File "/home/laura/.local/share/virtualenvs/jokusoramame-SR0dS4Ee/lib/python3.6/site-packages/curious/core/httpclient.py", line 393, in post
    return await self.request(("POST", bucket), method="POST", path=url, *args, **kwargs)
  File "/home/laura/.local/share/virtualenvs/jokusoramame-SR0dS4Ee/lib/python3.6/site-packages/curious/core/httpclient.py", line 267, in request
    response = await self._make_request(*args, **kwargs)
  File "/home/laura/.local/share/virtualenvs/jokusoramame-SR0dS4Ee/lib/python3.6/site-packages/curious/core/httpclient.py", line 224, in _make_request
    return await asks.request(*args, headers=headers, timeout=5, **kwargs)
  File "/home/laura/.local/share/virtualenvs/jokusoramame-SR0dS4Ee/lib/python3.6/site-packages/asks/base_funcs.py", line 29, in request
    r = await s.request(method, url=uri, **kwargs)
  File "/home/laura/.local/share/virtualenvs/jokusoramame-SR0dS4Ee/lib/python3.6/site-packages/asks/sessions.py", line 163, in request
    sock, r = await self.timeout_manager(timeout, req_obj)
  File "/home/laura/.local/share/virtualenvs/jokusoramame-SR0dS4Ee/lib/python3.6/site-packages/asks/sessions.py", line 188, in timeout_manager
    sock, r = await req_obj.make_request()
  File "/home/laura/.local/share/virtualenvs/jokusoramame-SR0dS4Ee/lib/python3.6/site-packages/asks/request_object.py", line 169, in make_request
    content_type, content_len, body = await self._formulate_body()
  File "/home/laura/.local/share/virtualenvs/jokusoramame-SR0dS4Ee/lib/python3.6/site-packages/asks/request_object.py", line 398, in _formulate_body
    body = await self._multipart(self.files)
  File "/home/laura/.local/share/virtualenvs/jokusoramame-SR0dS4Ee/lib/python3.6/site-packages/asks/request_object.py", line 470, in _multipart
    pkg_body = await self._file_manager(v)
  File "/home/laura/.local/share/virtualenvs/jokusoramame-SR0dS4Ee/lib/python3.6/site-packages/asks/request_object.py", line 495, in _file_manager
    async with asynclib.aopen(path, 'rb') as f:
  File "/home/laura/.local/share/virtualenvs/jokusoramame-SR0dS4Ee/lib/python3.6/site-packages/curio/file.py", line 139, in __aenter__
    self._fileobj = await run_in_thread(partial(open, *self._open_args, **self._open_kwargs))
  File "/home/laura/.local/share/virtualenvs/jokusoramame-SR0dS4Ee/lib/python3.6/site-packages/curio/workers.py", line 94, in run_in_thread
    return await worker.apply(callable, args, call_on_cancel)
  File "/home/laura/.local/share/virtualenvs/jokusoramame-SR0dS4Ee/lib/python3.6/site-packages/curio/workers.py", line 323, in apply
    return future.result()
  File "/home/laura/.local/share/virtualenvs/jokusoramame-SR0dS4Ee/lib/python3.6/site-packages/curio/workers.py", line 242, in result
    raise self._exception from None
  File "/home/laura/.local/share/virtualenvs/jokusoramame-SR0dS4Ee/lib/python3.6/site-packages/curio/workers.py", line 312, in run_callable
    future.set_result(func(*args))
ValueError: embedded null byte

Any interest in infiltrating requests/urllib3 development?

Hi all,

Obviously there's no obligation here at all, if you're having fun working on asks then you should do that :-). But I thought I should at least check whether there was any interest in taking the best parts of asks and bringing them to requests/urllib3 themselves?

The requests/urllib3 devs are interested in switching to an h11-based backend and supporting async, and I even started a little skunkworks project to explore making this happen for real:

njsmith/urllib3#1

...but I don't have time to do it all myself so I'm looking for anyone interested in helping :-)

How to use one cookie for all How to use one cookie for all ?

example:
In order to parse the site you need one set of cookies, for all requests with one connections, the first time you access the site, you get the necessary cookies within the session, and all the further requests go already with the necessary cookies - approx.
If we use several connections, then the first hits of all connections without cookies occur, how do we make use of one cookie for any number of connections ?

async def hsession_t_stateful(s):
r = await s.get()

@curio_run
async def session_stateful():
s = HSession(
'http://0.0.0.0:8080', persist_cookies=True, connections=1)
async with curio.TaskGroup() as g:
for _ in range(100):
await g.spawn(hsession_t_stateful(s))

How to do this for Session and Session
connections = 1 -> works
connections = 10 -> not working

Exception while closing sockets on `session.request()`

I'm not sure if this is a regression but I updated to the new version 1.5.11 and I'm getting an exception while using a 200 connection session to make 35K post requests.

It fails after a while, usually after roughly 10K or 15K requests.

The traceback ends with:

  File "(...redacted...)/main.py", line 90, in send_consumption
    response = await session.post(f'{BASE_URL}', data=DATA[i])
  File "(...redacted...)/asks/sessions.py", line 183, in request
    await sock.close()
AttributeError: 'SocketStream' object has no attribute 'close'

which refers to this line:

asks/asks/sessions.py

Line 183 in 04b2d03

await sock.close()

I used pdb to try to figure out what was going on, this is what I got right when the exception is thrown (I edited the sessions.py file and added a catch for AttributeError that triggers a breakpoint):

(Pdb) print(r.headers)
{'server': 'nginx/1.13.10', 'date': 'Fri, 25 May 2018 16:43:14 GMT', 'content-length': '0', 'connection': 'close', 'x-content-type-options': 'nosniff', 'strict-transport-security': 'max-age=31536000'}

(Pdb) dir(sock)
['__abstractmethods__', '__aenter__', '__aexit__', '__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__slots__', '__str__', '__subclasshook__', '__weakref__', '_abc_cache', '_abc_negative_cache', '_abc_negative_cache_version', '_abc_registry', '_active', '_send_conflict_detector', 'aclose', 'getsockopt', 'host', 'port', 'receive_some', 'send_all', 'send_eof', 'setsockopt', 'socket', 'wait_send_all_might_not_block']

(Pdb) vars(sock)
{'socket': <trio.socket.socket fd=18, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('127.0.0.1', 56820), raddr=('127.0.0.1', 8080)>, '_send_conflict_detector': <trio._util.ConflictDetector object at 0x7f81ba9ea4a8>, '_active': True, 'host': 'http://localhost:8080', 'port': '8080'}

The code I'm running is very similar to this:

async def send_message(session, i):    
    response = await session.post(f'{BASE_URL}', data=DATA[i])
    print(f'{i}: response: {response.status_code}')

async def send_all_messages(n):
    async with trio.open_nursery() as nursery:
        async with asks.Session(connections=200) as session:
            for i in range(n):
                nursery.start_soon(send_message, session, i)       

trio.run(send_all_messages, 35000)

Passing string to ask.get() with ;

Hi,
I am trying to use asks for OSRM server requests.

I am sending a list of urls to my server.

Small example:


 async def main():
  r = await asks.get('http://35.204.249.209:5000/route/v1/driving/-73.62818000,40.75193800;-73.6356926,40.7418803?overview=simplified')
  print(r)
      
      
curio.run(main())

results in a bad request, because the OSRM server only gets

[info] 15-03-2018 15:54:39 0.064683ms 54.89.111.179 - python-asks/1.3.10 400 /route/v1/driving/-73.62818000,40.75193800?overview=simplified

And if I replace the ; with something else the OSRM gets the full URL (but of course its not working)

async def main():
  r = await asks.get('http://35.204.249.209:5000/route/v1/driving/-73.62818000,40.75193800%TEST-73.6356926,40.7418803?overview=simplified')
  print(r)
      
      
curio.run(main())

results in [info] 15-03-2018 15:59:10 0.110487ms 54.89.111.179 - python-asks/1.3.10 400 /route/v1/driving/-73.62818000,40.75193800%TEST-73.6356926,40.7418803?overview=simplified

What is the problem with passing ; in an URL?

EDIT:

If I replace ; with %3B it works, is there an option for automatic encoding?

How to implement work through a proxy server?

How to implement work through a proxy server?
And the ability to change the proxy for each new conversion

import asks
import curio

async def example():
proxies = {
'http': '162.244.134.196:8080',
'https': '162.244.134.196:8080',
}
r = await asks.get('https://example.org', proxies=proxies)
print(r.content)

curio.run(example())

Timeout does not affect socket connect call

asks/asks/sessions.py

Line 141 in 2028482

sock = await self._grab_connection(url)

This line can hang, when connecting to a non-routable address (try 10.255.255.1 for example). It is not affected by the timeout argument.

Code Coverage (auth sparsely tested)

Hey. Ran this, but you left irc before I could tell you :p

With coverage you can see how many of your statements are covered by tests.

Name                           Stmts   Miss  Cover
--------------------------------------------------
asks/__init__.py                   5      0   100%
asks/_event_loop_wrappers.py      16     16     0%
asks/auth.py                      83     63    24%
asks/base_funcs.py                14      0   100%
asks/cookie_utils.py              30      0   100%
asks/errors.py                    12      0   100%
asks/req_structs.py               45     17    62%
asks/request_object.py           332     63    81%
asks/response_objects.py         122     19    84%
asks/sessions.py                 124     10    92%
asks/utils.py                     34     14    59%
--------------------------------------------------
TOTAL                            817    202    75%

Install coverage and pytest-cov to recreate with pytest --cov=asks.

AttributeError in StreamBody context manager

StreamBody.__aenter__ tries to access self.session._checked_out_sockets, which doesn't exist. As far as I can tell, this is the only occurrence of _checked_out_sockets in the whole project.

Running this minimal example:

$ cat | python
import asks, trio

async def main():
    r = await asks.get('https://google.com/', stream=True)
    async with r.body:
        pass

asks.init('trio')
trio.run(main)

results in:

Traceback (most recent call last):
  File "<stdin>", line 10, in <module>

[trio and other third party calls]

  File "<stdin>", line 6, in main
  File ".../lib/python3.6/site-packages/asks/response_objects.py", line 182, in __aenter__
    self.session._checked_out_sockets.remove(self.sock)
AttributeError: 'Session' object has no attribute '_checked_out_sockets'

convenience method `Response.raise_for_status()`

requests Response objects have an raise_for_status() method that I enjoy using.

http://docs.python-requests.org/en/master/api/#requests.Response.raise_for_status

Please implement this in asks.

KeyError: 'transfer-encoding'

Traceback (most recent call last):
  File "/home/laura/.local/share/virtualenvs/automoderator-nZfczLQS/lib/python3.6/site-packages/asks/request.py", line 549, in _catch_response
    if int(resp_data['headers']['content-length']) > 0:
  File "/home/laura/.local/share/virtualenvs/automoderator-nZfczLQS/lib/python3.6/site-packages/asks/req_structs.py", line 79, in __getitem__
    return self._store[key.lower()][1]
KeyError: 'content-length'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/laura/.local/share/virtualenvs/automoderator-nZfczLQS/lib/python3.6/site-packages/curio/kernel.py", line 826, in _run_coro
    trap = current._send(current.next_value)
  File "/home/laura/.local/share/virtualenvs/automoderator-nZfczLQS/lib/python3.6/site-packages/curio/task.py", line 96, in _task_runner
    return await coro
  File "/home/laura/dev/discord/automoderator/automoderator/core/engine.py", line 133, in _handle_rule
    await fn(ctx, **kwargs)
  File "/home/laura/dev/discord/automoderator/automoderator/core/engine.py", line 175, in action_delete_message
    await msg.delete()
  File "/home/laura/.local/share/virtualenvs/automoderator-nZfczLQS/lib/python3.6/site-packages/curious/dataclasses/message.py", line 231, in delete
    await self._bot.http.delete_message(self.channel.id, self.id)
  File "/home/laura/.local/share/virtualenvs/automoderator-nZfczLQS/lib/python3.6/site-packages/curious/http/httpclient.py", line 596, in delete_message
    data = await self.delete(url, "messages:{}".format(channel_id))
  File "/home/laura/.local/share/virtualenvs/automoderator-nZfczLQS/lib/python3.6/site-packages/curious/http/httpclient.py", line 378, in delete
    return await self.request(("DELETE", bucket), method="DELETE", path=url, *args, **kwargs)
  File "/home/laura/.local/share/virtualenvs/automoderator-nZfczLQS/lib/python3.6/site-packages/curious/http/httpclient.py", line 251, in request
    response = await self._make_request(*args, **kwargs)
  File "/home/laura/.local/share/virtualenvs/automoderator-nZfczLQS/lib/python3.6/site-packages/curious/http/httpclient.py", line 212, in _make_request
    return await self.session.request(*args, headers=headers, **kwargs)
  File "/home/laura/.local/share/virtualenvs/automoderator-nZfczLQS/lib/python3.6/site-packages/asks/sessions.py", line 156, in request
    sock, r = await req_obj.make_request()
  File "/home/laura/.local/share/virtualenvs/automoderator-nZfczLQS/lib/python3.6/site-packages/asks/request.py", line 203, in make_request
    response_obj = await self._request_io(req, req_body, hconnection)
  File "/home/laura/.local/share/virtualenvs/automoderator-nZfczLQS/lib/python3.6/site-packages/asks/request.py", line 240, in _request_io
    response_obj = await self._catch_response(hconnection)
  File "/home/laura/.local/share/virtualenvs/automoderator-nZfczLQS/lib/python3.6/site-packages/asks/request.py", line 553, in _catch_response
    if resp_data['headers']['transfer-encoding'] == 'chunked':
  File "/home/laura/.local/share/virtualenvs/automoderator-nZfczLQS/lib/python3.6/site-packages/asks/req_structs.py", line 79, in __getitem__
    return self._store[key.lower()][1]
KeyError: 'transfer-encoding'

Errors when a HTTP server sends neither a Content-Length nor a Transfer-Encoding.

Duplicate path in response.url

OK, now I am getting the odd URLs from

response = await session.get(initial_url_that_will_redirect)
print(response.url)

https://example.com/identity/login?signin=61a17533a7d449728bb90790e31dc186/identity/login?signin=61a17533a7d449728bb90790e31dc186'

The issue seems comes from here:

asks/asks/request_object.py

Line 530 in e1ecb40

'url': (self.uri[:-1] if self.uri.endswith('/')

Perhaps, self.host should be defined and the line should be something like:

'url': self.host + self.path

response headers are case-sensitive

request headers are case-insensitive, but

r = await asks.get('http://httpbin.org/headers', headers={'content-LENGTH': '0'})
j = r.json()
assert j['headers']['CONTENT-length'] == '0'

gives KeyError: 'CONTENT-length'

Add pre-defined headers option to session init

At suggestion of @SunDwarf, enable setting headders for all requests in a Session.

Docs not in sync with pypi release 0.2.1 (latest); version bump on pypi? Tracking tasks / changes?

Hey @theelous3 et al. Really cool package—I found out about it from one of the trio PRs. I'm excited to play around with it.

Unfortunately, as I was trying to get started, I ran into some trouble with invoking asks.init per the docs. However, it seems the version on pypi and the docs are not in sync right now.

I saw that some changes went in a couple hours ago via this commit. What's the timeline to get the trio and curio support released? Is a released blocked by #21?

I'm also wondering if there's a way I can keep a pulse on (or help with) changes coming down the pike. I really love what's going on here and would like to use it! :)

Thanks y'all!

Error with trio

I'm using Debian experimental with a virtualenv.

This is the error that I got:

`$ python
Python 3.6.2 (default, Aug 4 2017, 14:35:04)
[GCC 6.4.0 20170724] on linux
Type "help", "copyright", "credits" or "license" for more information.

import trio
import asks
asks.init('trio')
Traceback (most recent call last):
File "", line 1, in
File "/home/lorenzo/project/football_prediction/lib/python3.6/site-packages/asks/init.py", line 76, in init
_async_lib.aopen = trio.open_file
AttributeError: module 'trio' has no attribute 'open_file'
`

TypeError when decompressing streamed response

Using StreamBody as data in http_utils.decompress does not work, because the builtin tools expect a byte-like object.

StreamBody's journey through `decompress`:

asks/asks/response_objects.py

Lines 80 to 82 in 074b851

    
           decompressor = decompress(parse_content_encoding(content_encoding), 
        
                                     encoding) 
        
           r = decompressor.send(body)

asks/asks/http_utils.py

Lines 31 to 36 in 074b851

    
               data = yield data.decode(encoding, errors='replace') 
        
           else: 
        
               data = yield data 
        
           for compression in compressions: 
        
               if compression in _compression_mapping: 
        
                   data = _compression_mapping[compression](data)

Resulting TypeError:

$ cat | python
import asks, trio

async def main():
    r = await asks.get('https://google.com/', stream=True)
    assert r.headers.get('content-encoding')
    r.text

asks.init('trio')
trio.run(main)

Traceback (most recent call last):
  File "<stdin>", line 9, in <module>

[ trio and other external calls]

  File "<stdin>", line 6, in main
  File ".../lib/python3.6/site-packages/asks/response_objects.py", line 103, in text
    return self._decompress(self.body, self.encoding)
  File ".../lib/python3.6/site-packages/asks/response_objects.py", line 82, in _decompress
    r = decompressor.send(body)
  File ".../lib/python3.6/site-packages/asks/http_utils.py", line 36, in decompress
    data = _compression_mapping[compression](data)
  File "/usr/lib/python3.6/gzip.py", line 531, in decompress
    with GzipFile(fileobj=io.BytesIO(data)) as f:
TypeError: a bytes-like object is required, not 'StreamBody'

Partial methods has no docstrings

No option to ignore SSL/TLS certificate verification like request's verify=False

For local development it's useful to be able to ignore invalid certificates.
Requests allows this via a keyword argument to a request (verify=False). It's not ideal, it still prints warnings to the console which can only be disabled in a very ugly way, at least from what I recall.

I tried to do the same with an asks request, since the API is so similar, but no luck, I can't get a request to be made if the certificate fails verification.

I think it's a useful feature that doesn't seem hard to implement, if you agree but prefer to work on other features I can take a crack at it during the weekend and send you a pull request.

Clarify in the documentation that sessions should be used whenever doing large numbers of requests

As a new user of asks, I was running out of file descriptors/sockets because I didn't know I had to use sessions. More detail here, but I was basically under the impression that a simple asks.post() would close/clean a socket before finishing and the connection pooling offered by Sessions was mostly an optimization to avoid constantly creating/closing sockets.

The documentation does mention the use of sessions when making large numbers of requests but doesn't make it clear that it's pretty much a requirement to avoid running out of sockets, at least from my interpretation of these two pages:
https://asks.readthedocs.io/en/latest/index.html#a-bigger-little-example
https://asks.readthedocs.io/en/latest/a-look-at-sessions.html#

By the way, I loved how asks is pretty much a drop-in replacement for requests and how simple to use it is! :D

http_timeout test is flaky

on 6e18977,

debian:~/src/asks/tests$ pytest --verbose --tb=native
=============================================== test session starts ================================================
platform linux -- Python 3.6.1+, pytest-3.1.2, py-1.4.34, pluggy-0.4.0 -- /home/raylu/env/bin/python3.6
cachedir: ../.cache
rootdir: /home/raylu/src/asks, inifile:
collected 25 items 

test_asks.py::test_https_get PASSED
test_asks.py::test_bad_www_and_schema_get PASSED
test_asks.py::test_https_get_alt PASSED
test_asks.py::test_http_get PASSED
test_asks.py::test_http_redirect PASSED
test_asks.py::test_http_max_redirect_error PASSED
test_asks.py::test_http_max_redirect PASSED
test_asks.py::test_http_timeout_error PASSED
test_asks.py::test_http_timeout FAILED
test_asks.py::test_param_dict_set PASSED
test_asks.py::test_data_dict_set PASSED
test_asks.py::test_cookie_dict_send PASSED
test_asks.py::test_header_set PASSED
test_asks.py::test_file_send_single PASSED
test_asks.py::test_file_send_double PASSED
test_asks.py::test_file_and_data_send PASSED
test_asks.py::test_json_send PASSED
test_asks.py::test_gzip PASSED
test_asks.py::test_deflate PASSED
test_asks.py::test_chunked_te PASSED
test_asks.py::test_stream PASSED
test_asks.py::test_callback PASSED
test_asks.py::test_hsession_smallpool PASSED
test_asks.py::test_session_stateful PASSED
test_asks.py::test_Session_smallpool PASSED

===================================================== FAILURES =====================================================
________________________________________________ test_http_timeout _________________________________________________
Traceback (most recent call last):
  File "/home/raylu/src/asks/asks/sessions.py", line 156, in request
    timeout, response_task.join())
  File "/home/raylu/env/lib/python3.6/site-packages/curio-0.7-py3.6.egg/curio/task.py", line 730, in _timeout_after_func
    return await coro
  File "/home/raylu/env/lib/python3.6/site-packages/curio-0.7-py3.6.egg/curio/task.py", line 110, in join
    await self.wait()
  File "/home/raylu/env/lib/python3.6/site-packages/curio-0.7-py3.6.egg/curio/task.py", line 121, in wait
    await _scheduler_wait(self.joining, 'TASK_JOIN')
  File "/home/raylu/env/lib/python3.6/site-packages/curio-0.7-py3.6.egg/curio/traps.py", line 110, in _scheduler_wait
    yield (_trap_sched_wait, sched, state)
curio.errors.TaskTimeout: 34329.550673733

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/raylu/env/lib/python3.6/site-packages/curio-0.7-py3.6.egg/curio/kernel.py", line 835, in _run_coro
    trap = current._send(current.next_value)
  File "/home/raylu/env/lib/python3.6/site-packages/curio-0.7-py3.6.egg/curio/task.py", line 96, in _task_runner
    return await coro
  File "/home/raylu/src/asks/tests/test_asks.py", line 73, in test_http_timeout
    r = await asks.get('http://httpbin.org/delay/1', timeout=2)
  File "/home/raylu/src/asks/asks/base_funcs.py", line 29, in request
    r = await s.request(method, url=uri, **kwargs)
  File "/home/raylu/src/asks/asks/sessions.py", line 159, in request
    raise RequestTimeout
asks.errors.RequestTimeout

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/raylu/src/asks/tests/test_asks.py", line 11, in func_wrapper
    return curio.run(func(*args, **kwargs))
  File "/home/raylu/env/lib/python3.6/site-packages/curio-0.7-py3.6.egg/curio/kernel.py", line 882, in run
    return kernel.run(corofunc, *args, timeout=timeout)
  File "/home/raylu/env/lib/python3.6/site-packages/curio-0.7-py3.6.egg/curio/kernel.py", line 214, in run
    raise TaskError('Task Crashed') from ret_exc
curio.errors.TaskError: Task Crashed
----------------------------------------------- Captured stderr call -----------------------------------------------
Task 27 crashed
Traceback (most recent call last):
  File "/home/raylu/src/asks/asks/sessions.py", line 156, in request
    timeout, response_task.join())
  File "/home/raylu/env/lib/python3.6/site-packages/curio-0.7-py3.6.egg/curio/task.py", line 730, in _timeout_after_func
    return await coro
  File "/home/raylu/env/lib/python3.6/site-packages/curio-0.7-py3.6.egg/curio/task.py", line 110, in join
    await self.wait()
  File "/home/raylu/env/lib/python3.6/site-packages/curio-0.7-py3.6.egg/curio/task.py", line 121, in wait
    await _scheduler_wait(self.joining, 'TASK_JOIN')
  File "/home/raylu/env/lib/python3.6/site-packages/curio-0.7-py3.6.egg/curio/traps.py", line 110, in _scheduler_wait
    yield (_trap_sched_wait, sched, state)
curio.errors.TaskTimeout: 34329.550673733

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/raylu/env/lib/python3.6/site-packages/curio-0.7-py3.6.egg/curio/kernel.py", line 835, in _run_coro
    trap = current._send(current.next_value)
  File "/home/raylu/env/lib/python3.6/site-packages/curio-0.7-py3.6.egg/curio/task.py", line 96, in _task_runner
    return await coro
  File "/home/raylu/src/asks/tests/test_asks.py", line 73, in test_http_timeout
    r = await asks.get('http://httpbin.org/delay/1', timeout=2)
  File "/home/raylu/src/asks/asks/base_funcs.py", line 29, in request
    r = await s.request(method, url=uri, **kwargs)
  File "/home/raylu/src/asks/asks/sessions.py", line 159, in request
    raise RequestTimeout
asks.errors.RequestTimeout
======================================= 1 failed, 24 passed in 18.39 seconds =======================================
Task(id=97, name='session_t_smallpool', <coroutine object session_t_smallpool at 0x7f9e6c50f678>, state='TERMINATED') never joined

rerunning immediately passed

malformed data for https url

import multio, trio, asks

async def main():
    r = await asks.get('https://www.bitstamp.net/api/v2/trading-pairs-info/')
      
multio.init('trio')
trio.run(main)

Traceback (most recent call last):
  File "/Users/michael/.pyenv/versions/3.6.3/lib/python3.6/site-packages/asks/request_object.py", line 520, in _catch_response
    response = await self._recv_event(hconnection)
  File "/Users/michael/.pyenv/versions/3.6.3/lib/python3.6/site-packages/asks/request_object.py", line 588, in _recv_event
    event = hconnection.next_event()
  File "/Users/michael/.pyenv/versions/3.6.3/lib/python3.6/site-packages/h11/_connection.py", line 439, in next_event
    exc._reraise_as_remote_protocol_error()
  File "/Users/michael/.pyenv/versions/3.6.3/lib/python3.6/site-packages/h11/_util.py", line 71, in _reraise_as_remote_protocol_error
    raise self
  File "/Users/michael/.pyenv/versions/3.6.3/lib/python3.6/site-packages/h11/_connection.py", line 420, in next_event
    event = self._extract_next_receive_event()
  File "/Users/michael/.pyenv/versions/3.6.3/lib/python3.6/site-packages/h11/_connection.py", line 361, in _extract_next_receive_event
    event = self._reader(self._receive_buffer)
  File "/Users/michael/.pyenv/versions/3.6.3/lib/python3.6/site-packages/h11/_readers.py", line 174, in maybe_read_from_SEND_RESPONSE_server
    return class_(headers=list(_decode_header_lines(lines[1:])), **matches)
  File "/Users/michael/.pyenv/versions/3.6.3/lib/python3.6/site-packages/h11/_readers.py", line 115, in _decode_header_lines
    matches = validate(header_field_re, line)
  File "/Users/michael/.pyenv/versions/3.6.3/lib/python3.6/site-packages/h11/_util.py", line 91, in validate
    raise LocalProtocolError(msg)
h11._util.RemoteProtocolError: malformed data

Python 3.6.3, asks==1.3.10

Requires trio/curio but neither are listed in dependencies

Issue is as in title. They should be listed in dependencies, or the relevant wrappers imported inside functions to prevent failure when one is not installed.

Would it be hard to add support for python 3.5?

I'm looking at using asks in the trio tutorial, and it's a little awkward to be like "so before starting, make sure you have trio, which works on cpython 3.5+ or pypy3, except for this part of the tutorial, to run the examples using asks, cpython 3.5 and pypy3 don't work, you need cpython 3.6+". It's not a huge deal if it's hard to fix, but I thought I'd at least check :-).

From a quick skim I see one use of async generators in asks.response_objects.StreamBody, which looks easy enough to fix. Is there anything else?

HSession and DSession perfomance issues.

My terrible homebrew locks in the session classes are wreacking havoc. I am currently swapping them out for semaphores.

Initial testing with minor fixes shows a billion % increase in speed.

error

import asks
import curio

asks.init('curio')

async def example(url):
r = await asks.get(url)
print(r.content)

#not work
url = 'https://simferopol.drom.ru/opel/meriva/24011092.html'

#work
#url = 'https://simferopol.drom.ru'
curio.run(example(url))

Traceback (most recent call last):
File "/home/wku/anaconda3/lib/python3.6/site-packages/asks/request.py", line 524, in _catch_response
response = await self._recv_event(hconnection)
File "/home/wku/anaconda3/lib/python3.6/site-packages/asks/request.py", line 583, in _recv_event
event = hconnection.next_event()
File "/home/wku/anaconda3/lib/python3.6/site-packages/h11/_connection.py", line 439, in next_event
exc._reraise_as_remote_protocol_error()
File "/home/wku/anaconda3/lib/python3.6/site-packages/h11/_util.py", line 71, in _reraise_as_remote_protocol_error
raise self
File "/home/wku/anaconda3/lib/python3.6/site-packages/h11/_connection.py", line 420, in next_event
event = self._extract_next_receive_event()
File "/home/wku/anaconda3/lib/python3.6/site-packages/h11/_connection.py", line 361, in _extract_next_receive_event
event = self._reader(self._receive_buffer)
File "/home/wku/anaconda3/lib/python3.6/site-packages/h11/_readers.py", line 171, in maybe_read_from_SEND_RESPONSE_server
matches = validate(status_line_re, lines[0])
File "/home/wku/anaconda3/lib/python3.6/site-packages/h11/_util.py", line 91, in validate
raise LocalProtocolError(msg)
h11._util.RemoteProtocolError: malformed data

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/wku/anaconda3/lib/python3.6/site-packages/curio/kernel.py", line 826, in _run_coro
trap = current._send(current.next_value)
File "/home/wku/anaconda3/lib/python3.6/site-packages/curio/task.py", line 96, in task_runner
return await coro
File "/home/wku/Рабочий стол/[project]/[project_work]/[auto]/src/app//m.py", line 8, in example
r = await asks.get(url)
File "/home/wku/anaconda3/lib/python3.6/site-packages/asks/base_funcs.py", line 29, in request
r = await s.request(method, url=uri, **kwargs)
File "/home/wku/anaconda3/lib/python3.6/site-packages/asks/sessions.py", line 138, in request
sock, r = await req_obj.make_request()
File "/home/wku/anaconda3/lib/python3.6/site-packages/asks/request.py", line 203, in make_request
response_obj = await self._request_io(req, req_body, hconnection)
File "/home/wku/anaconda3/lib/python3.6/site-packages/asks/request.py", line 240, in _request_io
response_obj = await self._catch_response(hconnection)
File "/home/wku/anaconda3/lib/python3.6/site-packages/asks/request.py", line 527, in _catch_response
response = await self._recv_event(hconnection)
File "/home/wku/anaconda3/lib/python3.6/site-packages/asks/request.py", line 583, in _recv_event
event = hconnection.next_event()
File "/home/wku/anaconda3/lib/python3.6/site-packages/h11/_connection.py", line 418, in next_event
"Can't receive data when peer state is ERROR")
h11._util.RemoteProtocolError: Can't receive data when peer state is ERROR

...&response_type=code id_token token&...

asks followed the redirect but the server gave an error response.

Requests worked fine. When I compared, requests replaces spaces with %20 with
requote_uri.

	decompressor = decompress(parse_content_encoding(content_encoding),
	encoding)
	r = decompressor.send(body)

	data = yield data.decode(encoding, errors='replace')
	else:
	data = yield data
	for compression in compressions:
	if compression in _compression_mapping:
	data = _compression_mapping[compression](data)

theelous3 / asks Goto Github PK

asks's People

Contributors

Stargazers

Watchers

Forkers

asks's Issues

StreamBody's journey through decompress:

Resulting TypeError:

Recommend Projects

Recommend Topics

Recommend Org

StreamBody's journey through `decompress`: