Request - detailed steps using other LLM model than the default llama2 about genai-stack HOT 9 CLOSED

I think I corrected that in my setup, but I will have to check later and get back to you on that too. Thanks again!

from genai-stack.

All fetch should be done by the pull-model service and because the LLM variable is passed through as an environment variable, you shouldn't have to rebuild the image to update the LLM.

When you did up were the containers already down ? If not does --force-recreate works ?

from genai-stack.

Thanks for the swift reply. :-)

The containers were already down. I will have to try --force-recreate later and get back to you.

I just found https://www.youtube.com/watch?v=7VbCwkTJ1YA (6 min) called "Install Mistral on Docker Gen AI Stack Locally" by Fahd Mirza, but it is not a success story, Fahd was also not able to follow the instructions and pull in Mistral, i.e. I am not alone.

I have submitted a PR, as there are some differences between Linux distros, and I needed to change some things to allow the stack to get up and running on Debian GNU/Linux Unstable. Maybe that is Fahd's problem too? Perhaps it is possible to add different distros as profile options? I am not familiar with how that might be done, sorry.

Other than that, this is really cool, and I am looking forward to replacing the Stackoverflow API with our own APIs. :-) Thanks for all the hard work! :-)

from genai-stack.

The video you mentioned failed because host.docker.internal is not supported by Docker internal DNS on Linux. You need to change the OLLAMA_BASE_URL in the .env to http://llm:11434 (it has been updated in the README, but not at the time of the video).

Maybe the issue comes from that, can you show the error you have, it should be helpful ?

from genai-stack.

Right, so my .env file looks like below. If I change the LLM=llama2 to some other Ollama model's tag, should I also change EMBEDDING_MODEL from sentence_transformer to ollama?

#OPENAI_API_KEY=sk-...
#OLLAMA_BASE_URL=http://host.docker.internal:11434
OLLAMA_BASE_URL=http://llm:11434
NEO4J_URI=neo4j://database:7687
NEO4J_USERNAME=neo4j
NEO4J_PASSWORD=password
LLM=llama2 #or any Ollama model tag, or gpt-4 or gpt-3.5
EMBEDDING_MODEL=sentence_transformer #or openai or ollama

LANGCHAIN_ENDPOINT="https://api.smith.langchain.com"
LANGCHAIN_TRACING_V2=true # false
LANGCHAIN_PROJECT=#your-project-name
LANGCHAIN_API_KEY=#your-api-key ls_...

from genai-stack.

It believe it should work like that, what are the log from the output of docker-compose --profile linux up ?

from genai-stack.

When using the code from you pull-request, with LLM=llama2-uncensored:7b, I have something that seems to be working:
pull-model_1 | pulling ollama model llama2-uncensored:7b using http://llm:11434
pull-model_1 | pulling manifest
pull-model_1 | ... pulling model (0s) - will take several minutes
pull-model_1 | ... pulling model (10s) - will take several minutes
pull-model_1 | ... pulling model (20s) - will take several minutes

Do you have the same output ?

from genai-stack.

Hey, yes, I do get the same output when doing docker logs <containerid>. :-) It takes a long time to get the model, of course, so it might come down to a) lack of feedback early on when doing just "up" and b) impatience. Thank for all your help! 👍

pulling b5749cc827d3...  91% |████████████  | (3.5/3.8 G... pulling model (600s) - will take several minutes
... pulling model (610s) - will take several minutes
pulling b5749cc827... pulling model (620s) - will take several minutes         
... pulling model (630s) - will take several minutes
pulling b5749cc827d3...  97% |██████████████ | (3.7... pulling model (640s) - will take several minutes
... pulling model (650s) - will take several minutes
pulling b5749cc827d3... 100% |███████████████| (3.8/3.8 GB, 12 MB/s)           
pulling 8c17c2ebb0ea... 100% |█████████████████| (7.0/7.0 kB, 3.4 kB/s)        
... pulling model (660s) - will take several minutes
pulling 7c23fb36d801... 100% |█████████████████| (4.8/4.8 kB, 2.7 kB/s)        
... pulling model (670s) - will take several minutes
pulling 750599e5d655... 100% |██████████████████████| (40/40 B, 23 B/s)        
pulling 49257c2db461... 100% |███████████████████| (373/373 B, 222 B/s)        
verifying sha256 digest
... pulling model (680s) - will take several minutes
writing manifest
removing any unused layers
success

from genai-stack.

Yeah that's the issue with docker-compose up, I had to force the logs with docker-compose logs. You have all the logs when using docker compose up. Glad I could help!

from genai-stack.