skyscanner / aiotask-context Goto Github PK

View Code? Open in Web Editor NEW

159.0 159.0 25.0 39 KB

Contextual information for asyncio tasks

License: MIT License

Python 98.62% Makefile 1.38%

aiotask-context's People

Contributors

Stargazers

Watchers

aiotask-context's Issues

Add CHANGELOG and CONTRIBUTING

People is starting to use this package. Lets add a CHANGELOG and CONTRIBUTING files

Catch AttributeError when context does not exist

If the factory hasn't been set, we will see errors like

        if task:
>           return asyncio.Task.current_task().context.get(key, default)
E           AttributeError: '_asyncio.Task' object has no attribute 'context'

Instead of propagating this exception, an alternative way of dealing with this is:

logger.warning('task_factory for context module was not installed, returning default')
return default

In the set method, we would just show the warning.

This avoids impact when users are not aware of this module, forget to initialize the factory (or maybe they just don't want to), tests...

I'm not sure either this is something we want though, @pfreixes what do you think?

Add RTD integration

Lets document some stuff with docstrings

factories
set/get

Context is global for all the parallel coroutines running

This way if multiple concurrent web requests will set it, after setting they all will have same value on getting. I bet that the desired behavior is to share context between tasks in one 'thread', not all the tasks in the loop. Because the latter is not much better than just module-level global dict.

import asyncio

import aiotask_context as context


async def uno():
    context.set('x', 2)
    await asyncio.sleep(0.001)

    return context.get('x')


async def dos():
    context.set('x', 1)
    await asyncio.sleep(0.001)

    return context.get('x')


def test_results_are_various():
    loop = asyncio.get_event_loop()
    loop.set_task_factory(context.task_factory)  # This is the relevant line

    results = loop.run_until_complete(asyncio.gather(uno(), dos()))

    assert results[0] != results[1]

In this test ioloop will surely switch context between coros, because the io-like operation is present.

TypeError: 'Context' object does not support item assignment

@wraps(f)
async def decorated_function(self, request, *args, **kwargs):
     context.set("managers", Manager(None, None, None, None))
     return await f(self, request, *args, **kwargs)

Context.get raises an error if there is no event loop

https://github.com/Skyscanner/aiotask-context/blob/master/aiotask_context/__init__.py#L89

The doc string says that if there is no event loop, the default value will be returned.

The code however raises a ValueError.

Logging this bug, as I think the documented behavior is desired, not the current implementation.

def get(key, default=None):
    """
    Retrieves the value stored in key from the Task.context dict. If key does not exist,
    or there is no event loop running, default will be returned
...
    """
    if not asyncio.Task.current_task():
        raise ValueError(NO_LOOP_EXCEPTION_MSG.format(key))

    return asyncio.Task.current_task().context.get(key, default)

Support wait_for and others

wait_for support can be added by wrapping coroutines with "ensure_future". asyncio code already has the logic to support both futures and coroutines inside wait_for. Need to check for others

Please remove dependency to "pytest-runner"

Hi,

pytest-runner is deprecated, and breaks installing this library.

also see: https://github.com/pytest-dev/pytest-runner/

how to use with sanic mutiple worker

I used this package to add request_id when i use logging to record the access log. But i read the document, https://sanic.readthedocs.io/en/latest/sanic/deploying.html#asynchronous-support-and-sharing-the-loop, is this mean that aiotask-context not support mutiprocess? My english is poor, hope that i describe clear enough.

when use:

asyncio.set_event_loop(uvloop.new_event_loop())
    server = app.create_server(host="0.0.0.0", port=8000, return_asyncio_server=True)
    loop = asyncio.get_event_loop()
    loop.set_task_factory(context.task_factory)
    task = asyncio.ensure_future(server)
    try:
        loop.run_forever()
    except:
        loop.stop()

can you give a example how to use sanic mutiple workers?

Does it have integration with aiohttp using gunicorn?

I have an aiohttp web application which are deployed under gunicorn. So I am using gunicorn to start my application. I want to have a location in which I can store request ID. In that way, I can now use log without passing the request ID every time.

From what I understand, the loop's task factory should be replaced first with the factory that aiotask-context provided. After that, I can now store request ID and use it in logging.Filter.

It seems that I cannot set the task factory of the loop since creating the event loop is inside of the aiohttp.GunicornWebWorker. Is it possible that I can use aiotask-context without dropping the use of gunicorn?

Add test for count reference

Make sure keeping this context doesnt avoid objects to be garbace collected

Tests fail on Python 3.11

asyncio.coroutine has been removed from Python 3.11

$ python3.11 -mpytest
Test session starts (platform: linux, Python 3.11.0, pytest 7.1.2, pytest-sugar 0.9.6)
benchmark: 3.2.2 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /home/edward/src/2022/vendor/aiotask-context
plugins: benchmark-3.2.2, astropy-header-0.2.2, forked-1.4.0, flaky-3.7.0, anyio-3.6.2, sugar-0.9.6, hypothesis-6.36.0, doctestplus-0.12.1, kgb-7.1.1, repeat-0.9.1, django-4.5.2, timeout-2.1.0, pylama-7.4.3, cov-4.0.0, tornasync-0.6.0.post2, remotedata-0.3.3, mock-3.8.2, requests-mock-1.9.3, xdist-2.5.0, asyncio-0.19.0
asyncio: mode=Mode.STRICT
collecting ... 
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― ERROR collecting tests/test_acceptance.py ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
tests/test_acceptance.py:11: in <module>
    @asyncio.coroutine
E   AttributeError: module 'asyncio' has no attribute 'coroutine'

―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― ERROR collecting tests/test_context.py ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
tests/test_context.py:8: in <module>
    @asyncio.coroutine
E   AttributeError: module 'asyncio' has no attribute 'coroutine'

================================================================================== short test summary info ==================================================================================
FAILED tests/test_acceptance.py - AttributeError: module 'asyncio' has no attribute 'coroutine'
FAILED tests/test_context.py - AttributeError: module 'asyncio' has no attribute 'coroutine'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Results (0.13s):
$

Add tests for uvloop support

with a parametrized fixture would be enough

Python 3.7: PendingDeprecationWarning: Task.current_task() is deprecated, use asyncio.current_task() instead

I get a lot of deprecation warning spam in my unit tests, e.g.:

/home/olli/.pyenv/versions/3.7.0/envs/ml-microservice-3.7.0/lib/python3.7/site-packages/aiotask_context/__init__.py:57: PendingDeprecationWarning: Task.current_task() is deprecated, use asyncio.current_task() instead
  context = asyncio.Task.current_task(loop=loop).context

Task factory used in example

Is the simple task_factory used in the PoC example the best choice?

It seems you'd want the request id to propagate to child tasks created within request processing. So one of the other two task factories seem more appropriate. Likely the copying one.

Is that correct understanding? I wonder if the example should be updated.

Side note -- this functionality is super useful, and should be more deeply merged into asyncio IMO.

skyscanner / aiotask-context Goto Github PK

aiotask-context's People

Contributors

Stargazers

Watchers

Forkers

aiotask-context's Issues

Recommend Projects

Recommend Topics

Recommend Org