💡 idea: Ollama support about mattermost-plugin-ai HOT 7 CLOSED

@Twilek-de It works for me if I set the API URL to http://localhost:11434/v1 Maybe just add the /v1?
If that doesn't work for you please post the mattermost server logs and any logs from ollama.

from mattermost-plugin-ai.

@Twilek-de Just noted Ollama adding OpenAI compatibility: https://ollama.ai/blog/openai-compatibility
Does that work for you?

from mattermost-plugin-ai.

Oh good catch. Just fired up Ollama with their latest docker container. OpenAI compatible endpoint works fine when evoking it with curl. With Mattermost AI plugin nothing happens. There is no error message but the ollama container does not ramp up CPU time, as it does when it is running a query. Don´t know what is happing.

I have checked that Mattermost is really seeing the ollama container in the docker network, it does. My settings look like this.

Running curl http://ollama:11434/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "llama2","messages": [{"role": "system","content": "You are a helpful assistant."},{"role": "user","content": "Hello!"}]}' inside the mattermost container works just fine, so it is not connectivity.

Mattermost config.json looks like this:

"mattermost-ai": { "config": { "allowPrivateChannels": true, "allowedTeamIds": "chat", "enableLLMTrace": false, "enableUserRestrictions": false, "imageGeneratorBackend": "Ollama", "llmBackend": "Ollama", "onlyUsersOnTeam": "", "services": [ { "apiKey": "", "defaultModel": "llama2", "id": "j5dr1bnste", "name": "Ollama", "password": "", "serviceName": "openaicompatible", "url": "http://ollama:11434", "username": "" } ], "transcriptBackend": "Ollama" } },

from mattermost-plugin-ai.

Hmm Ollama supports a subset of the OpenAI API, namely completion. There is no support for streaming yet and I seem to remember that the MM AI Plugin uses the streaming endpoint....

Update:
Scratch that, the Ollama docu says streaming is supported as well...

from mattermost-plugin-ai.

Yes! That was it. Works now for me too. Thanks!

from mattermost-plugin-ai.

Hmm somehow the new version does time out. Is there any way to edit the timeout,,,
{
"caller": "app/plugin_api.go:1003",
"error": "timeout streaming",
"level": "error",
"msg": "Streaming result to post failed",
"plugin_id": "mattermost-ai",
"timestamp": "2024-03-12 22:22:33.939 Z"
}

from mattermost-plugin-ai.

Found the "streamingTimeoutSeconds" setting. Is working again now.

from mattermost-plugin-ai.