Still creating network traffic when using cached responses about aiohttp-client-cache HOT 11 CLOSED

I'll look into it. There is a good chance that I broke something recently. Like the Readme says, this project is very early in development!

from aiohttp-client-cache.

I'm not able to reproduce this locally. Are you running the latest version (0.1.4)? Can you post a complete code example?

And can you show me the output of this example?

import asyncio
from aiohttp_client_cache import CachedSession, SQLiteBackend

async def cache_test():
    async with CachedSession(cache=SQLiteBackend()) as session:
        for i in range(10):
            response = await session.get('http://httpbin.org/delay/1')
            print(f'Response {i}:', type(response).__name__)
        cached_responses = [v async for v in session.cache.responses.values()]
        print('Cached responses:', len(cached_responses))

asyncio.run(cache_test())

from aiohttp-client-cache.

Traceback (most recent call last): File "C:\Users\Will Sigg\Downloads\ghostdatabase\cachetest.py", line 12, in <module> asyncio.run(cache_test()) File "C:\Users\Will Sigg\AppData\Local\Programs\Python\Python38\lib\asyncio\runners.py", line 43, in run return loop.run_until_complete(main) File "C:\Users\Will Sigg\AppData\Local\Programs\Python\Python38\lib\asyncio\base_events.py", line 616, in run_until_complete return future.result() File "C:\Users\Will Sigg\Downloads\ghostdatabase\cachetest.py", line 5, in cache_test async with CachedSession(cache=SQLiteBackend()) as session: TypeError: 'NoneType' object is not callable [Finished in 0.6s]

from aiohttp-client-cache.

and to confirm: i am running 0.1.4

from aiohttp-client-cache.

SQLiteBackend requires aiosqlite. I could probably make a better error message for that when a dependency is missing.

Can you pip install aiosqlite (or pip install aiohttp-client-cache[backends] to get all optional dependencies) and try again?

from aiohttp-client-cache.

And here's an updated example that will detect how many times aiohttp.ClientSession._request actually gets called:

import asyncio
from aiohttp import ClientSession
from aiohttp_client_cache import CachedSession, CachedResponse, SQLiteBackend
from unittest.mock import patch

mock_response = CachedResponse('GET', None, 200, None, None)

@patch.object(ClientSession, '_request', side_effect=[mock_response])
async def cache_test(mock_request):
    async with CachedSession(cache=SQLiteBackend()) as session:
        for i in range(10):
            response = await session.get('http://httpbin.org/delay/3')
            print(f'Response {i}:', type(response).__name__)
        cached_responses =[v async for v in session.cache.responses.values()]
        print('Cached responses:', len(cached_responses))
    print(f'Real ClientSession._request() called {mock_request.call_count} times')

asyncio.run(cache_test())

from aiohttp-client-cache.

odd, that example you sent works, but task manager shows that my script is still using the network.

from aiohttp-client-cache.

Huh, well that's curious! With aiohttp.ClientSession._request mocked out, I'm not sure what would be creating network traffic. I don't have experience using Wireshark (although it's on my list of stuff to learn); would that be able to tell you what host(s) it's hitting and total bytes transferred?

from aiohttp-client-cache.

I tested this against a local Flask server, and it only hit the endpoint once. You can try it yourself with the Flask quickstart app, and run the first example above against http://localhost:5000/, and it should log only one request.

Whatever that network traffic is that you're seeing, it doesn't seem to be the result of fetching a response more than once. Let me know if you can get more details with Wireshark.

from aiohttp-client-cache.

I'll take another look at it. If you want, I can send some examples of how I init the cache in my scripts. I did check again today and it did still use the network so I have no idea why I cannot get it to work.

from aiohttp-client-cache.

@willsigg It's possible what you're seeing is just aiosqlite connections to the SQLite db. Using strace shows a bunch of EAGAIN (Resource temporarily unavailable) messages for each cache operation, maybe as a result of aiosqlite requesting non-blocking file operations. I will admit, though, that those low-level system calls are a bit over my head. I attached strace output using the example code above (also under examples) both with and without SQLite, so you can see the difference:

• strace with SQLite cache
• strace with non-persistent cache

One way or another, it clearly shows that only one real HTTP request is made to httpbin.org, with or without SQLite.

I think that's probably enough info to close this issue for now. What do you think?

from aiohttp-client-cache.