Comments (10)
All of this repo is python code, I have already run it on Ubuntu 22.04, and you just only need to fix some dependencies about it manually.
from chatrtx.
It will be définitively pertinent to have a version compatible with Ubuntu / Linux since there are scientificité work station with Linux on top.
from chatrtx.
Hi @noahc1510, can you describe the process you used to fix the dependencies?
Would be nice to add what you did to the readme so people can use it on linux.
You can follow this guide below to fix the requirement. I tested it on the official installer and created a repo about it: https://github.com/noahc1510/trt-llm-rag-linux
System Requirement
- Nvidia Driver:
sudo apt install nvidia-driver-535
- CUDA:
sudo apt install nvidia-cuda-toolkit
- NCCL:
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring_1.0-1_all.deb sudo dpkg -i cuda-keyring_1.0-1_all.deb sudo apt-get update sudo apt install libnccl2
- libmpi:
sudo apt install libopenmpi-dev
Installation
- Install miniconda, create new environment and install pytorch=2.1.0, mpi4py=3.1.5, tensorrt-llm
In China, you can use these command below without vpn:
conda create -n trtllm python=3.10 conda activate trtllm conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia conda install -c conda-forge mpi4py mpich pip install --no-cache-dir --extra-index-url https://pypi.nvidia.com tensorrt-llm
conda create -n trtllm python=3.10 conda activate trtllm conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia conda install -c conda-forge mpi4py mpich pip install -i https://pypi.tuna.tsinghua.edu.cn/simple --extra-index-url https://pypi.nvidia.com tensorrt-llm
- Install the requirements
pip install -r requirements.txt
from chatrtx.
I believe that the majority of users interested in a similar aspect are Linux users, and it's indeed disheartening to note that it's released without API and support for Linux OS. I hope they provide a Linux version soon.
from chatrtx.
Current application is only targeted for windows
from chatrtx.
Hi @noahc1510, can you describe the process you used to fix the dependencies?
Would be nice to add what you did to the readme so people can use it on linux.
from chatrtx.
@noahc1510 seconded. I would consider automating it and making a pr to add it. Basically throwing it against the wall, seeing if it sticks.
from chatrtx.
There are two points on the readme that border on insulting.
Firstly, it's aimed solely at Windows, when the vast majority of servers used to train ML models are on Linux. That's a bit much for a community that's been fighting for 30 years to explain why we use this OS.
And it's also getting very irritating to see a perpetual amalgam between Ubuntu, which is a distribution, and not the best at all, and GNU Linux.
It's time for a change of mentality once and for all.
Thank you, therefore, for making sure that these models and this kind of project are initially offered on GNU Linux in general.
Sorry for the irritation, but I've been in the business for 22 years and it's starting to get a bit desperate.
from chatrtx.
hi @noahc1510 . I have a ubuntu 23.10 running on RTX 4090, I've followed your instructions, after pip install -r requirements.txt I downloaded the engine and tokenizer files.
Now I'm getting a different error
(trtllm) vgtu@vgtu-Default-string:~/Tomo/trt-llm-rag-windows$ python app.py --trt_engine_path model/ --trt_engine_name llama_float16_tp1_rank0.engine --tokenizer_dir_path model/ --data_dir dataset/ Traceback (most recent call last): File "/home/vgtu/Tomo/trt-llm-rag-windows/app.py", line 26, in <module> from trt_llama_api import TrtLlmAPI #llama_index does not currently support TRT-LLM. The trt_llama_api.py file defines a llama_index compatible interface for TRT-LLM. File "/home/vgtu/Tomo/trt-llm-rag-windows/trt_llama_api.py", line 24, in <module> from llama_index.bridge.pydantic import Field, PrivateAttr ModuleNotFoundError: No module named 'llama_index.bridge'
at first it was only missing llama_index, installed it using pip install llama_index, and now llama_index.bridge is missing... Also tried using conda install llama_index, it installed, but this same error keeps showing.
Can you please help?
Thank you very much
from chatrtx.
yeah... I just found out that I had to change llama_index.bridge to llama_index.legacy.bridge , same for some others
from chatrtx.
Related Issues (20)
- garbage output ? HOT 1
- [TensorRT-LLM][ERROR] CUDA runtime error in cudaDeviceGetDefaultMemPool HOT 2
- not able to debug this even with gdb HOT 1
- Not able to generate engine on RTX4060 Laptop 8GB 100% CPU being utilised HOT 2
- default response language HOT 2
- How to switch language?! HOT 3
- What version of Cuda is needed? HOT 5
- Update to Llama 3 8B model HOT 3
- Dtype value read by app.py empty. Gets error message : Unsupported dtype HOT 2
- how do I define other large language models in the configuration?
- 'trtllm-build' is not recognized HOT 1
- ⁉️ Benefit of using ChatRTX instead of LMStudio or Ollama and other similar tools❓❔ HOT 2
- [Question] Would it be possible for the system to use spreadsheets as Dataset as well as .txt, .pdf, .doc HOT 1
- Importing from langchain will no longer be supported HOT 1
- already passed the verify_install.py but cant pass the app.py "ImportError: DLL load failed while importing tensorrt: 找不到指定的模块。" HOT 2
- Simplified version?
- code
- feature request - home assistant, replace siri HOT 2
- PDF Error? HOT 3
- Improvement suggestions HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from chatrtx.