Use typing in server/client about lmql HOT 4 CLOSED

Wow, thanks for getting right into it. I won't have much time to look into it over the weekend, but I will answer more concretely next week. Happy Easter to you as well.

dclib is an array-based library for implementing decoding algorithms independently from model backends and e.g. control flow logic. More on this, a bit more publicly, soon.

from lmql.

I've written synchronous version (I had problems with making the queues work) that can be used to illustrate the point.

from lmql.

Thanks for the work. Can you comment on how FastAPI compares to e.g. gRPC with respect to throughput and latency. We are currently planning to optimise the LMQL Inference API or even switch to an established solution altogether.

from lmql.

With the updated inference infrastructure, the API has been replaced by a socket-based custom protocol, i.e. LMTP: https://github.com/eth-sri/lmql/tree/main/src/lmql/models/lmtp

from lmql.