aio-libs / aiobotocore Goto Github PK

View Code? Open in Web Editor NEW

1.1K 26.0 175.0 1.03 MB

asyncio support for botocore library using aiohttp

Home Page: https://aiobotocore.rtfd.io

License: Apache License 2.0

Makefile 0.35% Python 99.65%

asyncio aiohttp botocore aws aws-sdk cloud-management cloud

aiobotocore's People

Contributors

Stargazers

Watchers

Forkers

bdrosen96 elastic-coders harai obmarg alanc10n oselivanov cbrichford jc2k luminger rymir ojii alfiyazi demmsnt vas3k jjonek vharitonsky hellysmile andriipetruk jreuter viachkakovskyi graingert achimnol alefteris ridha pfreixes dfee gitter-badger tomers devopsmi gridl kalaspuff ticosax jlcaro jostheim supergil82 alexanderbardin lexsca dalazx adamrothman gr1n alexmbird omrikiei oleksandr-kuzmenko yunstanford alex-berger qria jogusd seunguklee suburbanmtman e271828- superduper vir-mir imqqq stupoid mpacer willingc austinbarkley begatsby ssatia hypoport nathandemaria oba11 osaroai vvnkr alexanderlukanin13 liuzemeeting bhargavrpatel bsitruk philbasford maozihe jrbourbeau kevchentw viktor-havr craigmccarter msfernandes kingking888 joeledwards dazza-codes larryprice markshuang wing-plutoflume hlongmore potatochip davidbuniat igorsobreira codias jheiss pidoorrr meneal admdev8 cbuschka joseph-jones scriptautomate sacapuntas9 uplol gusostow geeker4py incrmntal fraglab miketzian

aiobotocore's Issues

Blocking call when loading credentials in Session.create_client

in aiobotocore.session.Session we make use of self.get_credentials that is implemented in botocore.session.Session.

This method's implementation depends on credential provider config, and in some cases it can block while doing IO, for instance when loading from ini file or when using a IAM role

head_object is broken

it will not return the ETag, the issue is due to #116. botocore expect the ETag to be in the correct case.

Got exception while receiving SQS messages

Hi, as I see yarl deps updated and I got to many exception...

Traceback (most recent call last):
  File "/Users/ng/Work/Projects/Python/packages/mspy/ms/core/transports/sqs.py", line 97, in _listen
    QueueUrl=queue_url, **receive_options)
  File "/Users/ng/.virtualenvs/venv42/lib/python3.6/site-packages/aiobotocore/client.py", line 78, in _make_api_call
    operation_model, request_dict)
  File "/Users/ng/.virtualenvs/venv42/lib/python3.6/site-packages/aiobotocore/endpoint.py", line 211, in _send_request
    exception)):
  File "/Users/ng/.virtualenvs/venv42/lib/python3.6/site-packages/aiobotocore/endpoint.py", line 243, in _needs_retry
    caught_exception=caught_exception, request_dict=request_dict)
  File "/Users/ng/.virtualenvs/venv42/lib/python3.6/site-packages/botocore/hooks.py", line 227, in emit
    return self._emit(event_name, kwargs)
  File "/Users/ng/.virtualenvs/venv42/lib/python3.6/site-packages/botocore/hooks.py", line 210, in _emit
    response = handler(**kwargs)
  File "/Users/ng/.virtualenvs/venv42/lib/python3.6/site-packages/botocore/retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "/Users/ng/.virtualenvs/venv42/lib/python3.6/site-packages/botocore/retryhandler.py", line 251, in __call__
    caught_exception)
  File "/Users/ng/.virtualenvs/venv42/lib/python3.6/site-packages/botocore/retryhandler.py", line 269, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/Users/ng/.virtualenvs/venv42/lib/python3.6/site-packages/botocore/retryhandler.py", line 317, in __call__
    caught_exception)
  File "/Users/ng/.virtualenvs/venv42/lib/python3.6/site-packages/botocore/retryhandler.py", line 223, in __call__
    attempt_number, caught_exception)
  File "/Users/ng/.virtualenvs/venv42/lib/python3.6/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
  File "/Users/ng/.virtualenvs/venv42/lib/python3.6/site-packages/aiobotocore/endpoint.py", line 269, in _get_response
    request.method, request.url, request.headers, request.body)
  File "/Users/ng/.virtualenvs/venv42/lib/python3.6/site-packages/aiobotocore/endpoint.py", line 200, in _request
    allow_redirects=False)
  File "/Users/ng/.virtualenvs/venv42/lib/python3.6/site-packages/aiohttp/client.py", line 577, in __iter__
    resp = yield from self._coro
  File "/Users/ng/.virtualenvs/venv42/lib/python3.6/site-packages/aiohttp/client.py", line 213, in _request
    proxy=proxy, proxy_auth=proxy_auth, timer=timer)
  File "/Users/ng/.virtualenvs/venv42/lib/python3.6/site-packages/aiohttp/client_reqrep.py", line 73, in __init__
    assert isinstance(url, URL), url
AssertionError: http://localhost:9324/

Any ideas?

Unclosed connection when reading data

I get example https://github.com/aio-libs/aiobotocore/blob/master/examples/simple.py
replace line 28 on

resp = yield from client.get_object(Bucket=bucket, Key=key)
data = yield from resp['Body'].readany()

And when i run this code i have message from aiohttp (i think):

Unclosed connection

In tests https://github.com/aio-libs/aiobotocore/blob/master/tests/test_basic_s3.py#L107 i see resp['Body'].close() But if i add this line i get error AttributeError: 'FlowControlStreamReader' object has no attribute 'close'

ImportError in client.py

Steps to reproduce:

create a fresh virtualenv: virtualenv -p /usr/local/bin/python3 ab

install aiobotocore:

./ab/bin/pip install --upgrade --no-cache-dir git+https://github.com/jettify/aiobotocore.git@master#egg=aiobotocore

Try to import aiobotocore: ./ab/bin/python -c 'import aiobotocore'

Output:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "~/ab/lib/python3.5/site-packages/aiobotocore/__init__.py", line 3, in <module>
    from .session import get_session, AioSession
  File "~/ab/lib/python3.5/site-packages/aiobotocore/session.py", line 6, in <module>
    from .client import AioClientCreator
  File "~/ab/lib/python3.5/site-packages/aiobotocore/client.py", line 2, in <module>
    from botocore.utils import get_service_module_name
ImportError: cannot import name 'get_service_module_name'

The root cause seems to be the fact that get_service_module_name is referenced despite not existing in botocore 1.1.7 which aiobotocore specifies in the install_requires variable of setup.py. This issue can be worked around by manually upgrading botocore to 1.2.2 or greater.

support waiters

It seems there's currently no support for creating waiters with aiobotocore clients (for example, see EC2 waiters: https://botocore.readthedocs.io/en/latest/reference/services/ec2.html#waiters)

Since botocore waiters necessarily block for a long time, it would be especially cool if aiobotocore could support them!

Find out how to test integration code on travis ci

Currently I need to execute tests manually on each PR, we need to find how to use amazon credentials on travis ci in safe way.

add to pip

Can this get added to PyPi so it can be pip installed?

upgrade botocore version

I think there's three tasks here until there's official asyncio support from boto

Do the manual process
create a script that will create an upgrade issue each time there's a new version of botocore detected
Investigate if we can automate this any more
- detect files added/removed
- detect new methods/signatures, function hashes, etc

Suggestion: Use localstack for testing

https://bitbucket.org/atlassian/localstack
This project emulates AWS APIs in the HTTP server level instead of mocking API client methods.
We could write more realistic test cases using this, though I haven't tried this personally yet.

Fix retry logic

see discussion in #44

move to python >=3.5

Do we want to move to python 3.5 only?

provide mypy type annotation

RuntimeError with aiobotocore + Tornado

Hopefully this is the right repo to post this issue to, but I suspect the issue may be from aiobotocore's calling of aiohttp.

The example code:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
from asyncio import get_event_loop
from datetime import timedelta

import aiobotocore
from tornado.gen import with_timeout, convert_yielded
from tornado.httpserver import HTTPServer
from tornado.platform.asyncio import AsyncIOMainLoop
from tornado.web import Application
from tornado.web import RequestHandler


class Handler(RequestHandler):

    async def get(self):
        session = aiobotocore.get_session()
        client = session.create_client('s3')

        response = await client.list_buckets()

        self.write(response)


if __name__ == '__main__':
    AsyncIOMainLoop().install()

    app = Application([
        (r'/', Handler),
    ])
    server = HTTPServer(app)
    server.listen(9999)

    get_event_loop().run_forever()

The full exception:

Uncaught exception GET / (127.0.0.1)
HTTPServerRequest(protocol='http', host='localhost:9999', method='GET', uri='/', version='HTTP/1.1', remote_ip='127.0.0.1', headers={'Accept': '*/*', 'User-Agent': 'curl/7.43.0', 'Host': 'localhost:9999'})
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/tornado/web.py", line 1469, in _execute
    result = yield result
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/tornado/gen.py", line 1015, in run
    value = future.result()
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/tornado/concurrent.py", line 237, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/tornado/gen.py", line 285, in wrapper
    yielded = next(result)
  File "<string>", line 6, in _wrap_awaitable
  File "test_aiobotocore.py", line 20, in get
    response = await client.list_buckets()
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/aiobotocore/client.py", line 78, in _make_api_call
    operation_model, request_dict)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/aiobotocore/endpoint.py", line 211, in _send_request
    exception)):
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/aiobotocore/endpoint.py", line 243, in _needs_retry
    caught_exception=caught_exception, request_dict=request_dict)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/botocore/hooks.py", line 227, in emit
    return self._emit(event_name, kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/botocore/hooks.py", line 210, in _emit
    response = handler(**kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/botocore/retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/botocore/retryhandler.py", line 251, in __call__
    caught_exception)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/botocore/retryhandler.py", line 269, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/botocore/retryhandler.py", line 317, in __call__
    caught_exception)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/botocore/retryhandler.py", line 223, in __call__
    attempt_number, caught_exception)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/aiobotocore/endpoint.py", line 269, in _get_response
    request.method, request.url, request.headers, request.body)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/aiobotocore/endpoint.py", line 200, in _request
    allow_redirects=False)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/aiohttp/client.py", line 577, in __iter__
    resp = yield from self._coro
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/aiohttp/client.py", line 200, in _request
    with timer:
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/aiohttp/helpers.py", line 750, in __enter__
    raise RuntimeError('Timeout context manager should be used '
RuntimeError: Timeout context manager should be used inside a task
500 GET / (127.0.0.1) 68.54ms

I have read both aio-libs/aiohttp#1180 and aio-libs/aiohttp#877 which are most probably related to this issue.

This example code is a modification of the working example (which I tested works on my system before implementing aiobotocore) provided in issue 1180.

I also tried adding the with_timeout and convert_yielded wrappers as suggested in 877 but I receive the same error.

I know that 1180 ended up in allowing timeout=None to be passed to a client request, perhaps something similar can be implemented here that eventually gets passed to the client session request?

System information:

Python 3.5.2
aiobotocore (0.2.1)
aiohttp (1.3.3)
botocore (1.5.0)
tornado (4.4.2)

Fix aiohttp 2.0.0 support

I did some testing here #135 for now aiohttp version is fixed.

Add tests for python 3.5 features like async with and async for

get_object is broken in case of concurrent execution.

I've tried to build simplest possible web server talking to s3 and found an issue:

Run following example:

#!/usr/bin/env python3
from os import environ
from time import time

import aiobotocore
from aiohttp import web
from asyncio import get_event_loop

bucket = 'my-bucket'
key = 'my-1mb-key'

session = aiobotocore.get_session()
client = session.create_client(
    's3', region_name='us-east-1',
    aws_secret_access_key=environ['AWS_SECRET_ACCESS_KEY'],
    aws_access_key_id=environ['AWS_ACCESS_KEY_ID'])

async def get_content(key):
    obj = await client.get_object(Bucket=bucket, Key=key)
    body = await obj['Body'].read()
    return body

async def index(request):
    print('got request')
    t = time()
    results = await get_content(key)
    content = str(len(results)).encode() + b' ' + str(time() - t).encode()
    print('request done')
    return web.Response(body=content)

app = web.Application()
app.router.add_route('GET', '/', index)

loop = get_event_loop()
handler = app.make_handler()
f = loop.create_server(handler, '0.0.0.0', 8080)
srv = loop.run_until_complete(f)
print('serving on', srv.sockets[0].getsockname())
try:
    loop.run_forever()
except KeyboardInterrupt:
    pass
finally:
    loop.run_until_complete(handler.finish_connections(1.0))
    srv.close()
    loop.run_until_complete(srv.wait_closed())
    loop.run_until_complete(app.finish())
loop.close()

Then use

    siege -c 4 -t 600s 'http://localhost:8080/'

You'll get following output:

** SIEGE 3.1.0
** Preparing 4 concurrent users for battle.
The server is now under siege...
HTTP/1.1 200   3.76 secs:      17 bytes ==> GET  /
HTTP/1.1 200   3.76 secs:      17 bytes ==> GET  /
HTTP/1.1 200   4.19 secs:      17 bytes ==> GET  /
HTTP/1.1 500   2.35 secs:     170 bytes ==> GET  /
HTTP/1.1 200   2.39 secs:      18 bytes ==> GET  /
HTTP/1.1 500   2.42 secs:     170 bytes ==> GET  /
HTTP/1.1 500   2.69 secs:     170 bytes ==> GET  /
HTTP/1.1 500   9.37 secs:     170 bytes ==> GET  /
HTTP/1.1 500   1.79 secs:     170 bytes ==> GET  /
HTTP/1.1 200   4.79 secs:      17 bytes ==> GET  /
HTTP/1.1 200   2.83 secs:      17 bytes ==> GET  /
HTTP/1.1 500   2.58 secs:     170 bytes ==> GET  /
HTTP/1.1 500   2.12 secs:     170 bytes ==> GET  /
HTTP/1.1 500   2.54 secs:     170 bytes ==> GET  /
HTTP/1.1 200   5.29 secs:      17 bytes ==> GET  /
HTTP/1.1 200   7.09 secs:      17 bytes ==> GET  /
HTTP/1.1 500   5.86 secs:     170 bytes ==> GET  /
HTTP/1.1 500   5.43 secs:     170 bytes ==> GET  /
HTTP/1.1 500   2.38 secs:     170 bytes ==> GET  /
HTTP/1.1 200   5.51 secs:      16 bytes ==> GET  /

In logs:

Error handling request
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/aiohttp/server.py", line 262, in start
    yield from self.handle_request(message, payload)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/aiohttp/web.py", line 87, in handle_request
    resp = yield from handler(request)
  File "./py35aio.py", line 25, in index
    results = await get_content(key)
  File "./py35aio.py", line 19, in get_content
    body = await obj['Body'].read()
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/aiohttp/streams.py", line 472, in wrapper
    result = yield from func(self, *args, **kw)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/aiohttp/streams.py", line 527, in read
    return (yield from super().read(n))
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/aiohttp/streams.py", line 247, in read
    block = yield from self.readany()
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/aiohttp/streams.py", line 472, in wrapper
    result = yield from func(self, *args, **kw)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/aiohttp/streams.py", line 535, in readany
    return (yield from super().readany())
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/aiohttp/streams.py", line 270, in readany
    yield from self._waiter
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/asyncio/futures.py", line 385, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/asyncio/tasks.py", line 288, in _wakeup
    value = future.result()
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
aiohttp.errors.ServerDisconnectedError
Error handling request
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/aiohttp/server.py", line 262, in start
    yield from self.handle_request(message, payload)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/aiohttp/web.py", line 87, in handle_request
    resp = yield from handler(request)
  File "./py35aio.py", line 25, in index
    results = await get_content(key)
  File "./py35aio.py", line 19, in get_content
    body = await obj['Body'].read()
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/aiohttp/streams.py", line 472, in wrapper
    result = yield from func(self, *args, **kw)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/aiohttp/streams.py", line 527, in read
    return (yield from super().read(n))
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/aiohttp/streams.py", line 247, in read
    block = yield from self.readany()
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/aiohttp/streams.py", line 472, in wrapper
    result = yield from func(self, *args, **kw)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/aiohttp/streams.py", line 535, in readany
    return (yield from super().readany())
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/aiohttp/streams.py", line 270, in readany
    yield from self._waiter
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/asyncio/futures.py", line 385, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/asyncio/tasks.py", line 288, in _wakeup
    value = future.result()
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
aiohttp.errors.ServerDisconnectedError

It seems that in case of multiple requests obj['Body'] might be broken.

    obj = await client.get_object(Bucket=bucket, Key=key)
    body = await obj['Body'].read()

Even if you try to move this block

session = aiobotocore.get_session()
client = session.create_client(
    's3', region_name='us-east-1',
    aws_secret_access_key=environ['AWS_SECRET_ACCESS_KEY'],
    aws_access_key_id=environ['AWS_ACCESS_KEY_ID'])

inside of the "get_content" function it will act the same.

document how to get back a file from s3

I've just helped somebody answering "how to get a file from s3 and iterate over it line by line" in stackoverflow there

http://stackoverflow.com/questions/40203620/aiobotocore-aiohttp-get-s3-file-content-and-stream-it-in-the-response/40204293#40204293

and it's only looking in the source code and this pull request #19
that's I've been able to answer it.

I think getting a file from s3 is common enough to see it in the readme. if the piece of code I provided in SO is clear enough, I will propose a PR in the readme

ImportError: cannot import name 'MultiDict'

Hi,

I get the following error when executing my service with Python 3.5.2:

Traceback (most recent call last):
  File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/vagrant/aiohttp-s3-files-manager/aiohttp-s3-files-manager/__main__.py", line 8, in <module>
    from .handle_get_connectivity_status import handle_get_connectivity_status
  File "/home/vagrant/aiohttp-s3-files-manager/aiohttp-s3-files-manager/handle_get_connectivity_status.py", line 5, in <module>
    import aiobotocore
  File "/tmp/virtual_env35/lib/python3.5/site-packages/aiobotocore/__init__.py", line 1, in <module>
    from .session import get_session, AioSession
  File "/tmp/virtual_env35/lib/python3.5/site-packages/aiobotocore/session.py", line 7, in <module>
    from .client import AioClientCreator
  File "/tmp/virtual_env35/lib/python3.5/site-packages/aiobotocore/client.py", line 11, in <module>
    from .args import AioClientArgsCreator
  File "/tmp/virtual_env35/lib/python3.5/site-packages/aiobotocore/args.py", line 9, in <module>
    from .endpoint import AioEndpointCreator
  File "/tmp/virtual_env35/lib/python3.5/site-packages/aiobotocore/endpoint.py", line 14, in <module>
    from aiohttp import MultiDict
ImportError: cannot import name 'MultiDict'

I simply import aiobotocore and asyncio in my file handle_get_connectivity_status.py.

My libraries versions:

aiobotocore (0.2.2)
aiohttp (2.0.0)
appdirs (1.4.3)
async-timeout (1.2.0)
botocore (1.5.26)
chardet (2.3.0)
docutils (0.13.1)
jmespath (0.9.2)
multidict (2.1.4)
packaging (16.8)
pip (9.0.1)
pyparsing (2.2.0)
python-dateutil (2.6.0)
setuptools (34.3.2)
six (1.10.0)
wheel (0.29.0)
yarl (0.10.0)

The requirements.txt I used to install my packages (as mentionned in the README.md of aiohttp and aiobotocore):

aiobotocore
aiohttp
async-timeout
chardet
multidict
yarl
botocore

Am I missing something or did I misundertand something ? ( I didn't get this problem before)
Thanks!

Unclosed response/connection errors

Here's a simplified version of the code I have for fetching all objects from an S3 bucket:

import os
import asyncio
import aiobotocore

session = aiobotocore.get_session()
client = session.create_client('s3')

async def fetch(summary):
    response = await client.get_object(
        Bucket=os.environ['S3_BUCKET_NAME'],
        Key=summary['Key'],
    )

    return await response['Body'].read()

async def main():
    response = await client.list_objects_v2(
        Bucket=os.environ['S3_BUCKET_NAME'],
    )

    for summary in response.get('Contents', []):
        body = await fetch(summary)

        print(summary['Key'], len(body))

loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.close()

client.close()

And here's some example output:

file_A.json 1250
file_B.json 38679
file_C.json 153
Unclosed response
client_response: <ClientResponse(https://example.s3.amazonaws.com/file_A.json) [200 OK]>
<CIMultiDictProxy('X-Amz-Id-2': 'xRqoQZjGSSIXPd1FlxhJ/hKdo68oNIwtfw1/KQbl3vNJmdTknBNgyFpYrqphpZ8q', 'X-Amz-Request-Id': 'BE5A3D402C597E6B', 'Date': 'Thu, 20 Oct 2016 23:11:43 GMT', 'Last-Modified': 'Fri, 16 Sep 2016 18:15:14 GMT', 'Etag': '"40167c63b5470c57781417407ab2f49e"', 'Accept-Ranges': 'bytes', 'Content-Type': 'binary/octet-stream', 'Content-Length': '1250', 'Server': 'AmazonS3')>

Unclosed response
client_response: <ClientResponse(https://example.s3.amazonaws.com/file_B.json) [200 OK]>
<CIMultiDictProxy('X-Amz-Id-2': 'MtPaOWwM5NEoYSifnv5Hw4JxSzMTIrgJ8JxOZNUDdDWRoCPd2Uh/51ftkRzmapde', 'X-Amz-Request-Id': 'BB83D803EF4AD0AD', 'Date': 'Thu, 20 Oct 2016 23:11:43 GMT', 'Last-Modified': 'Wed, 05 Oct 2016 17:52:26 GMT', 'Etag': '"d34947a3a4ee221de45ee15527a80d42"', 'Accept-Ranges': 'bytes', 'Content-Type': 'binary/octet-stream', 'Content-Length': '38679', 'Server': 'AmazonS3')>

Unclosed connection
client_connection: Connection<('example.s3.amazonaws.com', 443, True)>
Unclosed response
client_response: <ClientResponse(https://example.s3.amazonaws.com/file_C.json) [200 OK]>
<CIMultiDictProxy('X-Amz-Id-2': 'j55IXU/DfowTqYwH3caNiH2vUAsSA9gLJ+1oeAsGQOq82yUpmsg+7NNDEGgiDVds', 'X-Amz-Request-Id': '1483F994772ECB9B', 'Date': 'Thu, 20 Oct 2016 23:11:44 GMT', 'Last-Modified': 'Wed, 05 Oct 2016 16:17:19 GMT', 'Etag': '"833ec3055894c120f707dc5364781fbe"', 'Accept-Ranges': 'bytes', 'Content-Type': 'binary/octet-stream', 'Content-Length': '153', 'Server': 'AmazonS3')>

Why do I get those warnings? Do I need to use some sort of context manager?

make session a context manager

currently end up with:

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0xdeadbeef>

import contextlib

async def foo(loop):
    closing = contextlib.closing
    session = aiobotocore.get_session(loop=loop)
    with closing(session.create_client(cli_type, region_name='eu-west-1')) as c:
        return await bar(c)

add CONTRIBUTING.rst file to the root directory

We need to provide CONTRIBUTING.rst file with dev instruction to this repo.

New maintainers

@thehesiod @vas3k you did great job on supporting aiobotocore, I want to thank you for that!

If you still interested in this project I would like to invite you to aio-libs as aiobotocore maintainers. Rules are simple:

on each change please create pull request
each PR should be reviewed and tested
if PR is not commented for more than 3 days by other maintainers, feel free to merge it
be nice and respectful to others :)

What do you think? Feel free to ask me or @asvetlov (BDFL of aio-libs) any questions.

Error uploading file in us-west-2

Hello. I get an error when trying to upload to a bucket in us-west-2 region. This script works without issue if I upload to a bucket in us-east-1 :

Traceback (most recent call last):
  File "async-s3-2.py", line 28, in <module>
    event_loop.run_until_complete(main(event_loop, '/tmp/test_file'))
  File "/usr/lib/python3.5/asyncio/base_events.py", line 387, in run_until_complete
    return future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "async-s3-2.py", line 21, in main
    await write_to_s3(loop, f)
  File "async-s3-2.py", line 15, in write_to_s3
    Body=b'\x01'*1024)
  File "/usr/local/lib/python3.5/dist-packages/aiobotocore/client.py", line 91, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (TemporaryRedirect) when calling the PutObject operation: Please re-send this request to the specified temporary endpoint. Continue to use the original request endpoint for future requests.

Encountered this error with version '0.2.1'

Code is largely copied from example:

async def write_to_s3(loop, f):
  bucket = 'my-bucket'
  sess = aiobotocore.get_session(loop=loop)
  async with sess.create_client('s3', region_name='us-west-2',
    aws_secret_access_key='xxx,
    aws_access_key_id='xxx') as cli:
    resp = await cli.put_object(Bucket=bucket, 
                                   Key=f,
                                   Body=b'\x01'*1024)

Python 3.6 support

aiobotocore is currently dependent on botocore<=1.4.73 which has a compatibility problem with Python 3.6 - the fix is available in botocore>=1.4.87

Release aiobotocore 0.2.0

We have nice changes already, I believe we can release new version to PyPI

Example with custom S3 api server

Hi,

I have my own S3 api server, but I can't get data because cert can't verify. I changed verify=False in client it not solve an issue. Then I've seen code and found that is reading configuration from where? Also, I have my own certs bundle (ca_certs) I changed settings in client to read that, but i got same error with verification. Would you like to provide an example file with custom S3 api server connection?

Forcing content-type to 'application/octet-stream' breaks SQS requests.

I've been trying to get aiobotocore working with SQS, but was getting errors because the content-type being sent with the request was always set to application/octet-stream when SQS requires application/x-www-form-urlencoded.

I've managed to track this down to these lines https://github.com/jettify/aiobotocore/blob/master/aiobotocore/endpoint.py#L105-L112 that hard-code application/octet-stream. I assume these were needed somehow for S3, but they stop the usual botocore content-type code from working. As far as I can tell, the content-type would usually be set in https://github.com/boto/botocore/blob/082a8b8e564b12500ff3de4d048f89d56d33d67d/botocore/vendored/requests/models.py#L460

I've tried removing the application/octet-stream hard coding here and it does seem to fix SQS. I haven't thoroughly tested it though.

allow passing in an aiohttp ClientSession to allow host code to manage pooling etc.

vendor botocore

Currently aiobotocore and botocore are version locked, and new versions of botocore have been released.

This makes it impossible to install the latests versions of both awsclia and aiobotocore when using pip-compile (from pip-tools).

If you vendor (copy the source code into aiobotocore/_vendor/.../) it would be possible to use both in the same virtualenv without conflicts.

aiohttp 1.2+ ClientSession issues

ERROR [asyncio] Creating a client session outside of coroutine

due

https://github.com/KeepSafe/aiohttp/blob/c4539a7f3851483c985fa885f7e4a151e3bc3d43/aiohttp/client.py#L57

I can propose to implement lazy(on_demand) _aio_session initialization, if it is ok, I'll make pull request

Drop support for python 3.3 since aiohttp does not support this version

Cover all S3 API with simple test by first release

[Tracking] roll into botocore

just wondering, thanks!

HTTP Proxy settings not passed to aiohttp request

HTTP proxy defined in HTTP_PROXY and HTTPS_PROXY environment variables are not passed to aiohttp request, which makes the library unusable in private VPCs for other things than S3 with VPC endpoint.

Looks like the proxy configuration is read using utils derived from botocore but then it's just ignored in the AioEndpoint constructor.

aiohttp has moved MultiDict maybe import from multidict package instead?

Traceback (most recent call last):
  File "./redacted.py", line 4, in <module>
    import aiobotocore
  File "/opt/venv/lib/python3.5/site-packages/aiobotocore/__init__.py", line 1, in <module>
    from .session import get_session, AioSession
  File "/opt/venv/lib/python3.5/site-packages/aiobotocore/session.py", line 7, in <module>
    from .client import AioClientCreator
  File "/opt/venv/lib/python3.5/site-packages/aiobotocore/client.py", line 11, in <module>
    from .args import AioClientArgsCreator
  File "/opt/venv/lib/python3.5/site-packages/aiobotocore/args.py", line 9, in <module>
    from .endpoint import AioEndpointCreator
  File "/opt/venv/lib/python3.5/site-packages/aiobotocore/endpoint.py", line 14, in <module>
    from aiohttp import MultiDict
ImportError: cannot import name 'MultiDict'

aiobotocore will halt all aio coroutines on retry due to use of time.sleep

I've fixed this in #44

add body.set_socket_timeout(10) API

We need to support body.set_socket_timeout API like in botocor:

 257         response = self.client.get_object(
 258             Bucket=self.bucket_name, Key='foobarbaz')
 259         body = response['Body']
 260         # Am able to set a socket timeout
 261         body.set_socket_timeout(10)

Fix huge pagination logic

create_client fails

When i try to create a client, using the simple example for instance, I instantly get this error:

Traceback (most recent call last):
File "server/test_aws.py", line 36, in
loop.run_until_complete(go(loop))
File "/usr/local/lib/python3.5/asyncio/base_events.py", line 337, in run_until_complete
return future.result()
File "/usr/local/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/local/lib/python3.5/asyncio/tasks.py", line 239, in _step
result = coro.send(None)
File "server/test_aws.py", line 19, in go
aws_access_key_id=AWS_ACCESS_KEY_ID)
File "/cryptonuage/css/src/aiobotocore/aiobotocore/session.py", line 59, in create_client
client_config=config, api_version=api_version)
File "/usr/local/lib/python3.5/site-packages/botocore/client.py", line 69, in create_client
verify, credentials, scoped_config, client_config, endpoint_bridge)
TypeError: _get_client_args() takes 9 positional arguments but 10 were given

Am I doing anything wrong?

latest botocore breaks aiobotocore

   self._client = session.create_client(*args, **kwargs)
  File "/Users/amohr/dev/third_party/aiobotocore/aiobotocore/session.py", line 59, in create_client
    client_config=config, api_version=api_version)
  File "/usr/local/lib/python3.5/site-packages/botocore/client.py", line 69, in create_client
    verify, credentials, scoped_config, client_config, endpoint_bridge)
TypeError: _get_client_args() takes 9 positional arguments but 10 were given

looks like we need a fix or to fix the botocore version...long term we need a merge :)

Broken due to boto/botocore#936

Update aiobotocore to support botoctore >1.4.28

TypeError: init() missing 1 required positional argument: 'user_agent'

I got the following error when I tried to run the Basic Example:

Traceback (most recent call last):
  File "aios3.py", line 36, in <module>
    loop.run_until_complete(go(loop))
  File "/home/jharai/.pyenv/versions/3.5.0/lib/python3.5/asyncio/base_events.py", line 342, in run_until_complete
    return future.result()
  File "/home/jharai/.pyenv/versions/3.5.0/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/home/jharai/.pyenv/versions/3.5.0/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(value)
  File "aios3.py", line 19, in go
    aws_access_key_id=AWS_ACCESS_KEY_ID)
  File "/home/jharai/Sandbox/asyncio3.5/.async/lib/python3.5/site-packages/aiobotocore/session.py", line 49, in create_client
    client_config=config)
  File "/home/jharai/Sandbox/asyncio3.5/.async/lib/python3.5/site-packages/botocore/client.py", line 52, in create_client
    verify, credentials, scoped_config, client_config)
  File "/home/jharai/Sandbox/asyncio3.5/.async/lib/python3.5/site-packages/aiobotocore/client.py", line 48, in _get_client_args
    loop=self._loop)
  File "/home/jharai/Sandbox/asyncio3.5/.async/lib/python3.5/site-packages/aiobotocore/endpoint.py", line 168, in __init__
    super().__init__(endpoint_resolver, configured_region, event_emitter)
TypeError: __init__() missing 1 required positional argument: 'user_agent'

It seems that the number of arguments to botocore.endpoint.EndpointCreator doesn't match.

When I installed aiobotocore with pip install git+https://github.com/jettify/aiobotocore.git@master, botocore of version 0.109.0 was also installed as a dependency. Is this what is intended?

submit latest release to pypi

Should release the latest one to pypi, unfortunately I don't have perms ;)

Test botocore with fake AWS server

Fake server library:
https://github.com/spulec/moto

SimpleDB BatchDeleteAttributes Request is Malformed

I'm attempting to use aiobotocore to do batch_delete_attributes requests to SimpleDB, but the request is kicked back as malformed. A simple example that reproduces this based on the S3 example in the readme:

import asyncio
import aiobotocore

AWS_ACCESS_KEY_ID = "xxx"
AWS_SECRET_ACCESS_KEY = "xxx"

@asyncio.coroutine
def go(loop):

    domain = 'sdb-domain-name'
    item_name = '234cbd5f48f7418b9b059d4817bf40fe'

    session = aiobotocore.get_session(loop=loop)
    client = session.create_client('sdb', region_name='us-east-1',
                                   aws_secret_access_key=AWS_SECRET_ACCESS_KEY,
                                   aws_access_key_id=AWS_ACCESS_KEY_ID)

    resp = yield from client.batch_delete_attributes(
        DomainName=domain,
        Items=[{'Name': item_name}]
    )
    print(resp)


loop = asyncio.get_event_loop()
loop.run_until_complete(go(loop))

When run, this outputs:

$ python test_async.py
Traceback (most recent call last):
  File "test_async.py", line 27, in <module>
    loop.run_until_complete(go(loop))
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/asyncio/base_events.py", line 342, in run_until_complete
    return future.result()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(value)
  File "test_async.py", line 21, in go
    Items=[{'Name': item_name}]
  File "/Users/jpk/virtualenvs/sdb-whack/src/aiobotocore-master/aiobotocore/client.py", line 132, in _make_api_call
    raise ClientError(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidRequest) when calling the BatchDeleteAttributes operation: The request is malformed
Unclosed connector
connector: <aiohttp.connector.TCPConnector object at 0x110c172e8>
Fatal write error on socket transport
protocol: <asyncio.sslproto.SSLProtocol object at 0x110be9ba8>
transport: <_SelectorSocketTransport fd=10>
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/asyncio/selector_events.py", line 698, in write
    n = self._sock.send(data)
OSError: [Errno 9] Bad file descriptor
Fatal error on SSL transport
protocol: <asyncio.sslproto.SSLProtocol object at 0x110be9ba8>
transport: <_SelectorSocketTransport closing fd=10>
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/asyncio/selector_events.py", line 698, in write
    n = self._sock.send(data)
OSError: [Errno 9] Bad file descriptor

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/asyncio/sslproto.py", line 624, in _process_write_backlog
    self._transport.write(chunk)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/asyncio/selector_events.py", line 702, in write
    self._fatal_error(exc, 'Fatal write error on socket transport')
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/asyncio/selector_events.py", line 590, in _fatal_error
    self._force_close(exc)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/asyncio/selector_events.py", line 602, in _force_close
    self._loop.call_soon(self._call_connection_lost, exc)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/asyncio/base_events.py", line 453, in call_soon
    handle = self._call_soon(callback, args)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/asyncio/base_events.py", line 462, in _call_soon
    self._check_closed()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/asyncio/base_events.py", line 289, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
$

However, the same code using botocore synchronously works:

import asyncio
import aiobotocore
import botocore

AWS_ACCESS_KEY_ID = "xxx"
AWS_SECRET_ACCESS_KEY = "xxx"


def go():

    domain = 'sdb-domain-name'
    item_name = '234cbd5f48f7418b9b059d4817bf40fe'

    session = botocore.session.get_session()
    client = session.create_client('sdb', region_name='us-east-1',
                                   aws_secret_access_key=AWS_SECRET_ACCESS_KEY,
                                   aws_access_key_id=AWS_ACCESS_KEY_ID)

    resp = client.batch_delete_attributes(
        DomainName=domain,
        Items=[{'Name': item_name}]
    )
    print(resp)


go()

Which outputs:

{'ResponseMetadata': {'RequestId': '1ddaa063-51e4-3b4e-6257-10d3285bd61e', 'HTTPStatusCode': 200, 'BoxUsage': '0.0000219907'}}

For reference, the virtualenv this is running in looks like:

$ pip list
You are using pip version 6.1.1, however version 7.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
aiobotocore (0.0.1, /Users/jpk/virtualenvs/sdb-whack/src/aiobotocore-master)
aiohttp (0.18.3)
appnope (0.1.0)
botocore (1.1.7)
chardet (2.3.0)
decorator (4.0.4)
docutils (0.12)
gnureadline (6.3.3)
ipython (4.0.0)
ipython-genutils (0.1.0)
jmespath (0.7.1)
path.py (8.1.2)
pexpect (4.0.1)
pickleshare (0.5)
pip (6.1.1)
ptyprocess (0.5)
python-dateutil (2.4.2)
setuptools (15.0)
simplegeneric (0.8.1)
six (1.10.0)
traitlets (4.0.0)
$ python --version
Python 3.5.0
$

Am I doing something wrong, or is there a bug here?

S3 Put Object Failed with aiohttp==1.1.6

Hi,

I found a problem using aiobotocore with the latest version of aiohttp. Apparently, the latest aiohttp depends on yarl==0.8.1 or greater. If I downgrade it to aiohttp==1.1.5 with yarl==0.7.1, the problem doesn't occur.

Problem

If I'm using a parenthesis on Key value, it will throw an exception.

Example

import asyncio, aiobotocore
loop = asyncio.get_event_loop()
session = aiobotocore.get_session(loop=loop)
s3_client = session.create_client('s3')
s3_bucket = "test-bucket-name"
s3_path = "test-(key).txt"
loop.run_until_complete(s3_client.put_object(Bucket=s3_bucket,Key=s3_path, Body="test"))

The code throws:

botocore.exceptions.ClientError: An error occurred (SignatureDoesNotMatch) when calling the PutObject operation: The request signature we calculated does not match the signature you provided. Check your key and signing method.

However, the following code works properly:

import asyncio, aiobotocore
loop = asyncio.get_event_loop()
session = aiobotocore.get_session(loop=loop)
s3_client = session.create_client('s3')
s3_bucket = "test-bucket-name"
s3_path = "test-key.txt"
loop.run_until_complete(s3_client.put_object(Bucket=s3_bucket,Key=s3_path, Body="test"))

Tested with aiobotocore==0.0.6 and botocore==1.4.73

Port s3 pagination api to python 3.5 using async for statement

example doesn't work

When you try to run https://github.com/aio-libs/aiobotocore/blob/master/examples/simple.py on Python 3.5.1 it fails with:

Traceback (most recent call last):
  File "aiobot_test.py", line 36, in <module>
    loop.run_until_complete(go(loop))
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/asyncio/base_events.py", line 337, in run_until_complete
    return future.result()
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "aiobot_test.py", line 19, in go
    aws_access_key_id=AWS_ACCESS_KEY_ID)
  File "/Users/gav/.virtualenvs/pred-service/src/aiobotocore/aiobotocore/session.py", line 47, in create_client
    client_config=config, api_version=api_version)
  File "/Users/gav/.virtualenvs/pred-service/lib/python3.5/site-packages/botocore/client.py", line 65, in create_client
    verify, credentials, scoped_config, client_config)
  File "/Users/gav/.virtualenvs/pred-service/src/aiobotocore/aiobotocore/client.py", line 101, in _get_client_args
    self._get_signature_version_and_region(
AttributeError: 'AioClientCreator' object has no attribute '_get_signature_version_and_region'

Unfortunately, it doesn't look like a bug in example, but rather bug in a library.