Comments (3)
@rkoystart You can try docker run -e MODEL_ZOO="/models"
to add environment variable in the container. This variable is used to find .pb file when starting the custom engine.
from lightseq.
plz set environment variable MODEL_ZOO !
It seems that you forgot to set this environment variable required by the server.
i want the config.pbtxt and weight file for the transformer model that is supported by you if possible
You can check the proto/transformer.proto
file to generate your transformer model(.pb file) and the config.pbtxt
file only differs in names of inputs and outputs. Also, we'll consider adding a transformer example after the National Day holiday.
from lightseq.
@Taka152 even after initializing the environment variable i am getting the same error
(allennlp)nlp@nlp:/hdd1/ravi/ravi/lightseq$ export MODEL_ZOO="/hdd1/ravi/ravi/lightseq/modelzoo"
(allennlp)nlp@nlp:/hdd1/ravi/ravi/lightseq$ sudo docker run --gpus=1 --rm -p8000:8000 -p8001:8001 -p8002:8002 -v/hdd1/ravi/ravi/lightseq/modelzoo:/models
nvcr.io/nvidia/tensorrtserver:19.05-py3 trtserver --model-store=/models
from lightseq.
Related Issues (20)
- lightseq' Transformer expects an extra layer_norm on both encoder and decoder level
- LLaMA example 结果验证
- 请问lightseq在推理流程中有gemm调参这一步吗?
- Is llama inference available now? HOT 1
- Do you have plans to support token_type_ids?
- llama inference test HOT 3
- 请问lightseq可以支持segmentAnyting的推理优化吗 HOT 1
- 为什么连给的example也有bug? HOT 4
- [Question] gptj, mpt support.
- question about environment
- how to resolve xlm-roberta convert fail
- Can int8 in pre-training large model ???
- lightseq是否支持clip模型的int8量化?
- Is it normal that A10 inference speed is lower than 2080ti? HOT 1
- identifier "__hisnan" is undefined HOT 4
- 要求C++ 17 HOT 1
- Exception: Installed CUDA version 12.3 does not match the version torch was compiled with 12.1, unable to compile cuda/cpp extensions without a matching cuda version.
- 参数格式不对args format wrong
- is Baichuan2 model inference available now?
- [FeedBack][1.0.8][com.miniclip.eightballpool]- HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from lightseq.