Track allocated buffers in rpc-server about llama.cpp HOT 2 CLOSED

Should we make a dedicated session liked structure for each client to organize the resource allocatd by connection?

Right now the rpc-server can serve only one client at a time (I should add this to the README). I prefer to keep it that way because the code is simple as we don't have to deal with multiple threads, synchronization, etc. Users can still run multiple instances using the same backend and overcommit backend memory if this is what they want.

from llama.cpp.

Have some other thought maybe off topic:
Should we make a dedicated session liked structure for each client to organize the resource allocatd by connection? we can also hold ggml_backend_t there.

from llama.cpp.