Computationaly-intense search operations should be run in a thread about aiotinydb HOT 5 CLOSED

Is this still gonna happen, or is this a dead side project?

from aiotinydb.

Unfortunatelly, it's more of the latter. I intended to use this library in some personal projects, but they got postponed to a halt.

from aiotinydb.

Ok, looks like I'll have to make ZODB aio then

from aiotinydb.

As far as I can tell, a CPU-bound operation in a thread can still block the main event loop, so this requires multiprocessing.

The CPU-bound functions like insert(), search() etc. are implemented in the Table class, which is highly integrated into TinyDB and Storage. Implementing this would require writing a wrapper around Table that executes all methods in another process, which seems to be quite complicated (especially since AIOJSONStorage can't be pickled). And what if a user wants to use db.insert() in the main process, but db.search() should run in a different process? I think such an implementation is just not worth the effort.

With file locking implemented, it's possible to just use another instance of AIOTinyDB (or even TinyDB when taking care of filelocking yourself) and run it using a ProcessPoolExecutor:

def search(filename: str) -> list[Document]:
    async def _search():
        async with AIOTinyDB(filename) as db:
            return db.search(where("foo") == "bar")

    return asyncio.run(_search())

async def search_in_process_pool(filename: str) -> list[Document]:
    with concurrent.futures.ProcessPoolExecutor() as executor:
        return await asyncio.get_event_loop().run_in_executor(executor, search, filename)

Since AsyncIO is more commonly used to avoid performance issues caused by IO operations, I prefer to document that CPU-bound methods such as search() are blocking and provide an example that shows how multiprocessing can be used. What do you think @ASMfreaK?

(Edit: fixed the example)

from aiotinydb.

Very well written, thanks! We can now close this issue. You can also link it in docs and readme. Should someone come up with a new, better solution, then they should open a new issue.

from aiotinydb.