Code Monkey home page Code Monkey logo

llminator's People

Contributors

ashish-aesthisia avatar parveen232 avatar subhanshu0027 avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar

Forkers

subhanshu0027 cygwynd techthiyanes orefaleoluwayinka zeroxclem hercules261188 yacineali74 mdwoicke sontoriyama supersonic13 nymbo xjohnxjohn gopalgtm

llminator's Issues

running python webui.py gets me into console

Hi, I am trying to install and run this project, but after installing requirements and running "make", I am dropped into the console :
image
instead of running the server. I get no errors, everything installs smoothly (on a new cloud machine, in a fresh conda env with py=3.11).

Am I missing something? Thank you.

Initial execution stops at quantize_model

Hey,

I went by the readme and did the install (into a venv) and a make, but upon starting the webui the execution stops and returns a python interpreter prompt without getting to gradio or loading the model into memory.
image

I traced the stop point to this piece of code (quantizing of the default model):
image

llama.cpp was installed fine and I can run the convert-hf-to-gguf.py manually, so I'm a bit stumped as what might be the issue.

no errors, but not running

Hi there,

following the readme instructions I cloned and make the repo. Below are the logs and information, I hope useful.

The problem: The server is not running. It exits in the Python prompt

Expected: to have it running on localhost 7860

Any help will be appreciated.

Thank you

Node: I run this in vevn

Versions

lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.4 LTS
Release:	22.04
Codename:	jammy

uname -r
6.5.0-35-generic

python3 --version
Python 3.10.12
gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
pip --version
pip 22.0.2

Make

mkdir -p src/llama_cpp
git clone https://github.com/ggerganov/llama.cpp src/llama_cpp
Cloning into 'src/llama_cpp'...
remote: Enumerating objects: 25832, done.
remote: Counting objects: 100% (25831/25831), done.
remote: Compressing objects: 100% (7528/7528), done.
remote: Total 25832 (delta 18441), reused 25221 (delta 18077), pack-reused 1
Receiving objects: 100% (25832/25832), 46.28 MiB | 14.76 MiB/s, done.
Resolving deltas: 100% (18441/18441), done.
cd src/llama_cpp && \
	/home/ivo/Projects/Python/LLMinator/bin/pip3 install -r requirements.txt
Collecting numpy~=1.24.4
  Downloading numpy-1.24.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17.3/17.3 MB 40.7 MB/s eta 0:00:00
Collecting sentencepiece~=0.2.0
  Downloading sentencepiece-0.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 39.1 MB/s eta 0:00:00
Collecting transformers<5.0.0,>=4.40.1
  Downloading transformers-4.41.2-py3-none-any.whl (9.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.1/9.1 MB 49.8 MB/s eta 0:00:00
Collecting gguf>=0.1.0
  Downloading gguf-0.6.0-py3-none-any.whl (23 kB)
Collecting protobuf<5.0.0,>=4.21.0
  Downloading protobuf-4.25.3-cp37-abi3-manylinux2014_x86_64.whl (294 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 294.6/294.6 KB 26.5 MB/s eta 0:00:00
Requirement already satisfied: torch~=2.1.1 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from -r ./requirements/requirements-convert-hf-to-gguf.txt (line 2)) (2.1.2)
Collecting huggingface-hub<1.0,>=0.23.0
  Downloading huggingface_hub-0.23.2-py3-none-any.whl (401 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 401.7/401.7 KB 30.0 MB/s eta 0:00:00
Requirement already satisfied: tqdm>=4.27 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from transformers<5.0.0,>=4.40.1->-r ./requirements/requirements-convert-legacy-llama.txt (line 3)) (4.66.4)
Collecting tokenizers<0.20,>=0.19
  Downloading tokenizers-0.19.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.6/3.6 MB 45.2 MB/s eta 0:00:00
Requirement already satisfied: requests in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from transformers<5.0.0,>=4.40.1->-r ./requirements/requirements-convert-legacy-llama.txt (line 3)) (2.32.3)
Collecting safetensors>=0.4.1
  Downloading safetensors-0.4.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 40.9 MB/s eta 0:00:00
Requirement already satisfied: packaging>=20.0 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from transformers<5.0.0,>=4.40.1->-r ./requirements/requirements-convert-legacy-llama.txt (line 3)) (23.2)
Requirement already satisfied: pyyaml>=5.1 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from transformers<5.0.0,>=4.40.1->-r ./requirements/requirements-convert-legacy-llama.txt (line 3)) (6.0.1)
Requirement already satisfied: filelock in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from transformers<5.0.0,>=4.40.1->-r ./requirements/requirements-convert-legacy-llama.txt (line 3)) (3.14.0)
Collecting regex!=2019.12.17
  Downloading regex-2024.5.15-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (775 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 775.1/775.1 KB 29.5 MB/s eta 0:00:00
Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from torch~=2.1.1->-r ./requirements/requirements-convert-hf-to-gguf.txt (line 2)) (12.1.105)
Requirement already satisfied: fsspec in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from torch~=2.1.1->-r ./requirements/requirements-convert-hf-to-gguf.txt (line 2)) (2024.5.0)
Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from torch~=2.1.1->-r ./requirements/requirements-convert-hf-to-gguf.txt (line 2)) (8.9.2.26)
Requirement already satisfied: typing-extensions in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from torch~=2.1.1->-r ./requirements/requirements-convert-hf-to-gguf.txt (line 2)) (4.12.0)
Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from torch~=2.1.1->-r ./requirements/requirements-convert-hf-to-gguf.txt (line 2)) (12.1.0.106)
Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from torch~=2.1.1->-r ./requirements/requirements-convert-hf-to-gguf.txt (line 2)) (12.1.105)
Requirement already satisfied: networkx in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from torch~=2.1.1->-r ./requirements/requirements-convert-hf-to-gguf.txt (line 2)) (3.3)
Requirement already satisfied: jinja2 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from torch~=2.1.1->-r ./requirements/requirements-convert-hf-to-gguf.txt (line 2)) (3.1.4)
Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from torch~=2.1.1->-r ./requirements/requirements-convert-hf-to-gguf.txt (line 2)) (12.1.3.1)
Requirement already satisfied: triton==2.1.0 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from torch~=2.1.1->-r ./requirements/requirements-convert-hf-to-gguf.txt (line 2)) (2.1.0)
Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from torch~=2.1.1->-r ./requirements/requirements-convert-hf-to-gguf.txt (line 2)) (12.1.105)
Requirement already satisfied: nvidia-nccl-cu12==2.18.1 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from torch~=2.1.1->-r ./requirements/requirements-convert-hf-to-gguf.txt (line 2)) (2.18.1)
Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from torch~=2.1.1->-r ./requirements/requirements-convert-hf-to-gguf.txt (line 2)) (11.4.5.107)
Requirement already satisfied: sympy in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from torch~=2.1.1->-r ./requirements/requirements-convert-hf-to-gguf.txt (line 2)) (1.12.1)
Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from torch~=2.1.1->-r ./requirements/requirements-convert-hf-to-gguf.txt (line 2)) (11.0.2.54)
Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from torch~=2.1.1->-r ./requirements/requirements-convert-hf-to-gguf.txt (line 2)) (10.3.2.106)
Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from torch~=2.1.1->-r ./requirements/requirements-convert-hf-to-gguf.txt (line 2)) (12.1.105)
Requirement already satisfied: nvidia-nvjitlink-cu12 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from nvidia-cusolver-cu12==11.4.5.107->torch~=2.1.1->-r ./requirements/requirements-convert-hf-to-gguf.txt (line 2)) (12.5.40)
Requirement already satisfied: MarkupSafe>=2.0 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from jinja2->torch~=2.1.1->-r ./requirements/requirements-convert-hf-to-gguf.txt (line 2)) (2.1.5)
Requirement already satisfied: urllib3<3,>=1.21.1 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from requests->transformers<5.0.0,>=4.40.1->-r ./requirements/requirements-convert-legacy-llama.txt (line 3)) (2.2.1)
Requirement already satisfied: certifi>=2017.4.17 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from requests->transformers<5.0.0,>=4.40.1->-r ./requirements/requirements-convert-legacy-llama.txt (line 3)) (2024.2.2)
Requirement already satisfied: charset-normalizer<4,>=2 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from requests->transformers<5.0.0,>=4.40.1->-r ./requirements/requirements-convert-legacy-llama.txt (line 3)) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from requests->transformers<5.0.0,>=4.40.1->-r ./requirements/requirements-convert-legacy-llama.txt (line 3)) (3.7)
Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from sympy->torch~=2.1.1->-r ./requirements/requirements-convert-hf-to-gguf.txt (line 2)) (1.3.0)
Installing collected packages: sentencepiece, safetensors, regex, protobuf, numpy, huggingface-hub, gguf, tokenizers, transformers
  Attempting uninstall: numpy
    Found existing installation: numpy 1.26.4
    Uninstalling numpy-1.26.4:
      Successfully uninstalled numpy-1.26.4
  Attempting uninstall: huggingface-hub
    Found existing installation: huggingface-hub 0.21.1
    Uninstalling huggingface-hub-0.21.1:
      Successfully uninstalled huggingface-hub-0.21.1
Successfully installed gguf-0.6.0 huggingface-hub-0.23.2 numpy-1.24.4 protobuf-4.25.3 regex-2024.5.15 safetensors-0.4.3 sentencepiece-0.2.0 tokenizers-0.19.1 transformers-4.41.2
mkdir -p src/quantized_model
echo "py_cmd = /home/ivo/Projects/Python/LLMinator/bin/python" >> configs/config.ini

webui

python3 webui.py 
/home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages/langchain/__init__.py:29: UserWarning: Importing PromptTemplate from langchain root module is no longer supported. Please use langchain_core.prompts.PromptTemplate instead.
  warnings.warn(
/home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages/huggingface_hub/file_download.py:1194: UserWarning: `local_dir_use_symlinks` parameter is deprecated and will be ignored. The process to download files to a local folder has been updated and do not rely on symlinks anymore. You only need to pass a destination folder as`local_dir`.
For more details, check out https://huggingface.co/docs/huggingface_hub/main/en/guides/download#download-files-to-local-folder.
  warnings.warn(
README.md: 100%|███████████████████████████████████████████████████████████████████| 6.72k/6.72k [00:00<00:00, 15.6MB/s]
generation_config.json: 100%|███████████████████████████████████████████████████████████| 132/132 [00:00<00:00, 713kB/s]
LICENSE: 100%|█████████████████████████████████████████████████████████████████████| 7.45k/7.45k [00:00<00:00, 35.2MB/s]
config.json: 100%|█████████████████████████████████████████████████████████████████████| 738/738 [00:00<00:00, 3.70MB/s]
model.safetensors.index.json: 100%|████████████████████████████████████████████████| 29.4k/29.4k [00:00<00:00, 23.5MB/s]
.gitattributes: 100%|██████████████████████████████████████████████████████████████| 1.65k/1.65k [00:00<00:00, 6.02MB/s]
special_tokens_map.json: 100%|█████████████████████████████████████████████████████████| 587/587 [00:00<00:00, 3.50MB/s]
tokenizer_config.json: 100%|███████████████████████████████████████████████████████| 9.35k/9.35k [00:00<00:00, 30.7MB/s]
tokenizer.json: 100%|██████████████████████████████████████████████████████████████| 2.12M/2.12M [00:00<00:00, 2.94MB/s]
model-00002-of-00002.safetensors: 100%|██████████████████████████████████████████████| 610M/610M [01:07<00:00, 9.10MB/s]
stable-code-3b-q4_k_m.gguf: 100%|██████████████████████████████████████████████████| 1.71G/1.71G [01:16<00:00, 22.4MB/s]
stable-code-3b-q5_k_m.gguf: 100%|██████████████████████████████████████████████████| 1.99G/1.99G [02:10<00:00, 15.3MB/s]
model-00001-of-00002.safetensors: 100%|████████████████████████████████████████████| 4.98G/4.98G [02:52<00:00, 28.9MB/s]
Fetching 13 files: 100%|████████████████████████████████████████████████████████████████| 13/13 [02:52<00:00, 13.31s/it]
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux█████████████████| 4.98G/4.98G [02:52<00:00, 58.2MB/s]
Type "help", "copyright", "credits" or "license" for more information.█████████████| 1.99G/1.99G [02:10<00:00, 9.60MB/s]
>>>

example index.html

index.html:22 WebSocket connection to 'ws://localhost:7861/' failed: 
(anonymous) @ index.html:22
index.html:28 WebSocket is already in CLOSING or CLOSED state.
sendMessage @ index.html:28

others

python3 webui.py 
/home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages/langchain/__init__.py:29: UserWarning: Importing PromptTemplate from langchain root module is no longer supported. Please use langchain_core.prompts.PromptTemplate instead.
  warnings.warn(
/home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages/huggingface_hub/file_download.py:1194: UserWarning: `local_dir_use_symlinks` parameter is deprecated and will be ignored. The process to download files to a local folder has been updated and do not rely on symlinks anymore. You only need to pass a destination folder as`local_dir`.
For more details, check out https://huggingface.co/docs/huggingface_hub/main/en/guides/download#download-files-to-local-folder.
  warnings.warn(
Fetching 13 files: 100%|█████████████████████████████████████████████████████████████| 13/13 [00:00<00:00, 11962.69it/s]
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> exit
Use exit() or Ctrl-D (i.e. EOF) to exit
>>> exit()
/home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages/gradio/components/dropdown.py:93: UserWarning: The `max_choices` parameter is ignored when `multiselect` is False.
  warnings.warn(
/home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages/gradio/components/dropdown.py:179: UserWarning: The value passed into gr.Dropdown() is not in the list of choices. Please update the list of choices to include: stabilityai/stable-code-instruct-3b or set allow_custom_value=True.
  warnings.warn(
Traceback (most recent call last):
  File "/home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages/langchain_community/llms/llamacpp.py", line 140, in validate_environment
    from llama_cpp import Llama, LlamaGrammar
ImportError: cannot import name 'LlamaGrammar' from 'llama_cpp' (/home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages/llama_cpp/__init__.py)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ivo/Projects/Python/LLMinator/LLMinator/webui.py", line 202, in <module>
    llm_chain, llm = init_llm_chain(model_path)
  File "/home/ivo/Projects/Python/LLMinator/LLMinator/webui.py", line 48, in init_llm_chain
    llm = LlamaCpp(
  File "/home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages/pydantic/v1/main.py", line 339, in __init__
    values, fields_set, validation_error = validate_model(__pydantic_self__.__class__, data)
  File "/home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages/pydantic/v1/main.py", line 1100, in validate_model
    values = validator(cls_, values)
  File "/home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages/langchain_community/llms/llamacpp.py", line 142, in validate_environment
    raise ImportError(
ImportError: Could not import llama-cpp-python library. Please install the llama-cpp-python library to use this embedding model: pip install llama-cpp-python
IMPORTANT: You are using gradio version 4.27.0, however version 4.29.0 is available, please upgrade.
--------

ImportError: Could not import llama-cpp-python library

Running webui.py after fixing #42 yields:
image

This is also on a fresh system, after installing requirements and make-ing without errors. Also tried in a fresh conda env, and on the base python default in the ubuntu machine.

pip install llama-cpp-python says it's already installed. Any thoughts? Thanks

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.