Error when run LlamaIndex_Mistral7B-RAG.ipynb in Jupyter Notebook. about llamaindex-rag-linux-cuda HOT 17 OPEN

My issue is llama-index failure after conda install pytorch. And solved by install anything by pip only.

from llamaindex-rag-linux-cuda.

Thanks for sharing your experience @linuaries, that could be tried - just a note that the recent LlamaIndex libraries have been split into legacy and core so the code won't run as is. I am looking into updating it and will update the repository with a version that works then.

@chanwitkepha, the error does look like a cuda library issue, I'm wondering if there's a difference between cuda v12.1 which you have and version v12.2 which I have? Is it possible to try to update nvidia cuda drivers, as per nvidia-smi to v12.2? The conda list showed version of cuda libraries to be 12.2 so this may be an issue with incompatible libraries (12.2) vs driver (12.1).

from llamaindex-rag-linux-cuda.

Thanks @chanwitkepha, please try 12.2 so it matches the libraries

from llamaindex-rag-linux-cuda.

Additional information.

LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/local/cuda-12.1/lib64:/usr/local/cuda-11.8/lib64:

 ls -l /usr/local/

total 56
drwxr-xr-x  3 root root 12288 ม.ค.   5 21:23 bin
lrwxrwxrwx  1 root root    22 มิ.ย.  28  2023 cuda -> /etc/alternatives/cuda
lrwxrwxrwx  1 root root    25 ก.ค.  27  2023 cuda-11 -> /etc/alternatives/cuda-11
drwxr-xr-x 15 root root  4096 ก.ค.  27  2023 cuda-11.8
lrwxrwxrwx  1 root root    25 มิ.ย.  28  2023 cuda-12 -> /etc/alternatives/cuda-12
drwxr-xr-x 15 root root  4096 มิ.ย.  28  2023 cuda-12.1
drwxr-xr-x  2 root root  4096 มี.ค.   1  2018 etc
drwxr-xr-x  2 root root  4096 มี.ค.   1  2018 games
drwxr-xr-x  9 root root  4096 เม.ย. 13  2021 include
drwxr-xr-x 12 root root 12288 ธ.ค.   9  2022 lib
lrwxrwxrwx  1 root root     9 พ.ค.  11  2018 man -> share/man
drwxr-xr-x  2 root root  4096 มี.ค.   1  2018 sbin
drwxr-xr-x 15 root root  4096 ต.ค.  21 19:41 share
drwxr-xr-x  2 root root  4096 มี.ค.   1  2018 src

nvcc --version

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Mon_Apr__3_17:16:06_PDT_2023
Cuda compilation tools, release 12.1, V12.1.105
Build cuda_12.1.r12.1/compiler.32688072_0

from llamaindex-rag-linux-cuda.

I try again by install pytorch stable 2.2
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
And test again, It's not work

ImportError                               Traceback (most recent call last)
Cell In[6], line 1
----> 1 import torch
      3 from llama_index.llms import LlamaCPP
      4 from llama_index.llms.llama_utils import messages_to_prompt, completion_to_prompt

File ~/anaconda3/envs/LlamaIndexRAGLinux/lib/python3.12/site-packages/torch/__init__.py:27
     24     return sys.modules.get("torch._meta_registrations", None) is object
     26 from ._utils import _import_dotted_name, classproperty
---> 27 from ._utils import _functionalize_sync as _sync
     28 from ._utils_internal import get_file_path, prepare_multiprocessing_environment, \
     29     USE_RTLD_GLOBAL_WITH_LIBTORCH, USE_GLOBAL_DEPS
     31 # TODO(torch_deploy) figure out how to freeze version.py in fbcode build

ImportError: cannot import name '_functionalize_sync' from 'torch._utils' (/home/devteam/anaconda3/envs/LlamaIndexRAGLinux/lib/python3.12/site-packages/torch/_utils.py)

So I went back to current version in file environment.yml

Please help, Thank you.

from llamaindex-rag-linux-cuda.

Hi @chanwitkepha, would you be able to run conda list so we can see the versions and I can compare with mine?

from llamaindex-rag-linux-cuda.

Hi @chanwitkepha, would you be able to run conda list so we can see the versions and I can compare with mine?

Here is the list of packages that show with conda list.

LlamaIndexRAGLinux-COnda-List.txt

from llamaindex-rag-linux-cuda.

Thanks @chanwitkepha, is it possible to run the following in your condo environment to downgrade from python 3.12 to 3.11?

conda install python=3. 11

I have seen an issue that pytorch is about to support 3.12 but not yet. Though I don't know why it is working for me. I will also double check but in the meantime if you can try 3.11 please.

from llamaindex-rag-linux-cuda.

Ok I will try it again.

from llamaindex-rag-linux-cuda.

Thanks @chanwitkepha, is it possible to run the following in your condo environment to downgrade from python 3.12 to 3.11?

conda install python=3. 11

I have seen an issue that pytorch is about to support 3.12 but not yet. Though I don't know why it is working for me. I will also double check but in the meantime if you can try 3.11 please.

After trying conda install python=3.11 It have many conflicts then conda cannot install it.

Could you change your environment.yml file for compatible with python 3.11 so I can try to install conda environment again? Thank you.

from llamaindex-rag-linux-cuda.

Hi @chanwitkepha, I've just run the Mistral 7B notebook with the environment as it was (Python 3.12.1 and PyTorch 2.1.0) and it ran okay. I am running it in Visual Studio Code and selecting the Conda environment as the kernel.

I was wondering if it is a Nvidia CUDA library error - can you run nvidia-smi and see what version shows? Also, what card are you using?

from llamaindex-rag-linux-cuda.

This is output from my nvidia-smi and nvcc --version.

from llamaindex-rag-linux-cuda.

I run on Jupyter Notebook

from llamaindex-rag-linux-cuda.

Thanks for sharing your experience @linuaries, that could be tried - just a note that the recent LlamaIndex libraries have been split into legacy and core so the code won't run as is. I am looking into updating it and will update the repository with a version that works then.

@chanwitkepha, the error does look like a cuda library issue, I'm wondering if there's a difference between cuda v12.1 which you have and version v12.2 which I have? Is it possible to try to update nvidia cuda drivers, as per nvidia-smi to v12.2? The conda list showed version of cuda libraries to be 12.2 so this may be an issue with incompatible libraries (12.2) vs driver (12.1).

Thank you, I will upgrade this old machine from Ubuntu 18.04.6 to 22.04 + CUDA 12.2 (or 12.3) and try again.

from llamaindex-rag-linux-cuda.

Hi @chanwitkepha (and @linuaries) - I'm close to finishing the update to the latest llama-index version (core) and that includes an update to llama-cpp-python as well.

It is advised, by llama-index, that a brand new environment is created and I did this and installed the nvidia cuda toolkit (Base Installer instructions here: https://developer.nvidia.com/cuda-downloads

This is now Cuda v12.3.

I will update the environment.yml file as well.

It is working for me and hopefully will work for you, too. Please wait for the update over the next hour before cloning.

from llamaindex-rag-linux-cuda.

Hi @chanwitkepha (and @linuaries) - I'm close to finishing the update to the latest llama-index version (core) and that includes an update to llama-cpp-python as well.

It is advised, by llama-index, that a brand new environment is created and I did this and installed the nvidia cuda toolkit (Base Installer instructions here: https://developer.nvidia.com/cuda-downloads

This is now Cuda v12.3.

I will update the environment.yml file as well.

It is working for me and hopefully will work for you, too. Please wait for the update over the next hour before cloning.

Hi, I use git pull and try it again, It seem this environment.yml can work with CUDA 12.1 in my old Ubuntu 18.04.6 LTS.

Now I can pass step 3 but still have error in step 8. It seem my First GPU have too small memory,

But my PC have 2 GPU (GTX 1080) How can I use both GPU for processing this Mistral 7B Models.

from llamaindex-rag-linux-cuda.

Hi @chanwitkepha, it may be trying to load too many layers onto the GPU. For this n_gpu_layers value, try setting it lower, such as 10 and then if that leaves you a lot of VRAM (check with nvidia-smi) then you can increase it.

Note: The last line visible in my screenshot above says it found 1 CUDA device, I would think you should see this line and it should say 2 CUDA devices. So, I'm not 100% sure if it is using your cards correctly.

from llamaindex-rag-linux-cuda.