Comments (13)
@JWCook, certainly! Just tested at ae63533 and the process exited as expected.
from aiohttp-client-cache.
Thanks, I'm glad you've found it useful! I'll try to get at least a short-term fix for this released within the next couple weeks.
One thing that would help would be testing out the changes in #189, and let me know if you have any thoughts about that approach.
from aiohttp-client-cache.
This looks like an issue stemming from aiosqlite.Connection
which inherits from threading.Thread
.
from aiohttp-client-cache.
Could you give the changes in main
a try for me? I believe that will fix the issue. I'll get a patch release out for that soon.
from aiohttp-client-cache.
For those interested, the fix was introduced in a33331f.
from aiohttp-client-cache.
Fixed in 0.9.0
from aiohttp-client-cache.
Related to #147.
from aiohttp-client-cache.
I am reopening this because it seems that the switch to aiosqlite
has altered the supported semantics for using aiohttp_client_cache.CachedSession
. The following code hangs:
import asyncio
from aiohttp_client_cache import SQLiteBackend, CachedSession
async def main():
cache = SQLiteBackend()
client = CachedSession(cache=cache)
req = await client.get("http://httpbin.org/get")
resp = await req.text()
req.close()
print(resp)
# does not hang if properly close (identical behavior to async context manager)
# await client.close()
if __name__ == "__main__":
asyncio.run(main())
TLDR; the above code hangs because client.close()
is never called and awaited.
It is required that client.close()
is called and awaited because SQLiteBackend
creates and uses a aiosqlite.Connection
that runs in its own thread. The aiosqlite.Connection
acts like an event loop (see its run()
method). It accepts work (coroutines) over a queue and then reschedules coroutines on the main thread's event loop. This is the problem, in the above code, the event loop runs for a shorter amount of time than the aiosqlite.Connection
thread (as soon as main()
returns, the event loop starts its shutdown). This makes it impossible to close unless done implicitly or explicitly at the call site. In some cases adding a finalizer (__del__
) to SQLiteBackend
that schedules a close()
task on the main event loop to could remedy this, but there is no guarantee that the event loop will be open long enough for that task to be scheduled and run.
The unfortunate side-affect of this behavior is that all code that uses a SQLiteBackend
passes the responsibility of handling the lifetime of the aiosqlite.Connection
's thread to the end user. Meaning users of that code need to either explicitly call a shutdown method (i.e. close()
) or use the object via a context manager. For example:
import asyncio
from aiohttp_client_cache import SQLiteBackend, CachedSession
class Client:
def __init__(self):
self._session = CachedSession(cache=SQLiteBackend())
async def get(self) -> str:
async with self._session.get("http://httpbin.org/delay/1") as req:
return await req.text()
async def close(self) -> None:
# failing to call close will result in the program hanging
await self._session.close()
async def __aenter__(self) -> "Client":
return self
async def __aexit__(self, exc_type, exc_val, exc_tb) -> None:
await self.close()
async def main():
client = Client()
res = await client.get()
print(res)
# this hangs because client.close() is never called
if __name__ == "__main__":
asyncio.run(main())
from aiohttp-client-cache.
Are you still seeing this behavior in the latest release (0.9.1)? SQLite connections are now closed automatically on contextmanager exit, and your code snippet appears to work fine for me locally and in CI. Note that this behavior can also be changed with the autoclose
parameter (see #162).
from aiohttp-client-cache.
If the CachedSession is not properly closed this can happen, however, close is not called magically without using a contextmanager.
You have to do:
async with CachedSession(cache=SQLiteBackend()) as client:
# use client
after the with block finished, the session and the associated cache will be properly closed and no dangling sqlite connection will be running.
from aiohttp-client-cache.
Okay, I understand now. Yeah, getting aiosqlite to close properly on its own without using a contextmanager is a bit tricky. Looks like there are a couple relevant open issues on the aiosqlite repo, but there are some (less than ideal) ways I could hack around this in the mean time.
I'll make a separate issue for this (#187).
from aiohttp-client-cache.
@JWCook thanks for continuing to look into this and try and find a viable way to move forward. I am more than happy to collaborate on this issue, meaning submitting a PR, working through proposed solutions, or just testing it. I've been using your package for a long time and would like to continue to do so. You've always been quick to respond and great to communicate with!
from aiohttp-client-cache.
@JWCook for sure! I'll take a look at that over the next few days.
from aiohttp-client-cache.
Related Issues (20)
- `expire_after` actually doesn't remove key from cache backend (RedisBackend). Is this intended? HOT 4
- Make python-forge an optional/dev dependency only
- Drop support for python 3.7 HOT 2
- Add response filtering options
- CacheMixin doesn't pass kwargs to ClientSession when Sentry is used HOT 5
- .content instance is the same across multiple cached responses (for in-memory backend) HOT 2
- Feature request: support `fsspec` for `FileBackend` HOT 2
- python-forge, markdown-it-py dependencies are still mandatory HOT 3
- HTTP responses larger than 400kB are not handeled by DynamoDBBackend HOT 2
- Unit test fixes for FileBackend
- Feature request: FileBackend should honor the cache_name as base directory for all caches (responses and redirects)
- CachedSession does not correctly bypass the cache when caching is disabled via session.disabled()
- Automatically close SQLite connection without using contextmanager HOT 2
- Feature request: Update badge to show build status of main branch HOT 1
- AttributeError: 'CachedResponse' object has no attribute 'closed'
- Add support for python 3.12
- Add more conditional request tests
- Update ".github/workflows/build.yml" with dependabot HOT 3
- Switch to ruff for linting & formatting
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aiohttp-client-cache.