moonshot-admin / moonshot Goto Github PK

For locally developed models / most models downloaded from HuggingfaceHub, it can be troublesome to set up an API endpoint for the model. There are libraries such as vLLM that only extends support for models of certain architecture, or transformers-openai-api, which is not actively maintained. This means that it is even more difficult to benchmark custom models via Moonshot, which only supports model calls via API endpoints.

It would be a useful feature for researchers/developers to be able to use Moonshot as a library for benchmarking locally running models instead of solely for hosted models via API endpoints, given that the model can handle appropriately crafted prompts and provide a parseable text output.

During model development, it can prove useful to have a safe exit sequence for a currently running task to make quick changes to the model and redeploy, instead of force quitting the interactive Moonshot environment on CLI.

With the current interactive CLI environment, we have to open a new terminal in order to run a new task (cookbook/recipe) concurrently. Would it be possible to run them in such a way that they can be managed from the same terminal? Either in a session-based manner (e.g. tmux) or sending it to the background (e.g. nohup).