Comments (11)
yuyi1111
commented on May 30, 2024
1
6B模型输入长度200K需要大概多少显存呢？
from yi.
wangye01inf
commented on May 30, 2024
1
@yuyi1111 请问有安装 flash attention 吗？这个对显存影响挺大的
from yi.
excellent-ai
commented on May 30, 2024
Duplicate of #55
from yi.
FrankWhh
commented on May 30, 2024
这不是同一个问题
from yi.
learninmou
commented on May 30, 2024
demo中给出了一个text_generation_tp.py样例，如果是输入长度200K，在不开启offload技术且不量化的情况下，34B模型推理需要400GB以上的显存，我们测试使用是8卡A100/A800。
from yi.
FrankWhh
commented on May 30, 2024
硬算啊！显存占用400G，那速度呢？
from yi.
wangye01inf
commented on May 30, 2024
@yuyi1111 2TP 情况下 BS=1 每张卡大约需要 60-80GB 左右显存
from yi.
yuyi1111
commented on May 30, 2024
@yuyi1111 2TP 情况下 BS=1 每张卡大约需要 60-80GB 左右显存
能麻烦请教一下200K推理的情况吗？我用的text_generation_tp.py进行推理，两张80G显存的a800还是报cuda oom
from yi.
yuyi1111
commented on May 30, 2024
flash attention
谢谢！安装了之后就能跑起来了
from yi.
happy15
commented on May 30, 2024
6B模型输入长度200K需要大概多少显存呢？
同问，mbp 14 with 16g上通过ollama跑没跑起来。
from yi.
learninmou
commented on May 30, 2024
6B模型输入长度200K需要大概多少显存呢？
同问，mbp 14 with 16g上通过ollama跑没跑起来。
你好，没有在mbp上测试过，理论上需要NVIDIA的GPU。
from yi.
Related Issues (20)
- 就不能在官网挂一个体验的链接吗 HOT 1
- 6b运行正常，34b-int4运行失败 HOT 3
- 用示例代码，返回值不符合预期。 HOT 1
- 200K 上下文的模型什么时候能放出 chat 模型呢？ HOT 4
- How to inference in a multi-batch way? HOT 1
- 运行Yi-34B-Chat-4bits内核报错 HOT 1
- 用 VLLM 加载 Yi-34B-Chat-4bits-GPTQ, 模型有时不停地输出空字串而不停止 HOT 7
- 请问yi-34B-chat的模型支持更长的上下文吗？ HOT 1
- RuntimeError: probability tensor contains either `inf`, `nan` or element < 0 HOT 1
- yi-34b预训练模型支持多GPU分布参数进行量化和微调吗？ HOT 1
- The train/eval data format is different between the finetune guide and the chat model's chat template HOT 2
- sft报错：ValueError: YiForCausalLM does not support Flash Attention 2.0 yet. HOT 5
- 请教微调Yi-34B-Chat模型时context长度问题 HOT 1
- 微调 YI-6B 一直出现 loss scale overflow 然后 reduce 到 min_loss_scale 报错, YI-6B-Chat 则没问题，chat 模型训练参数设置有什么不同吗 HOT 4
- Function Calling 功能支持计划
- 参考finetune的quickstart，但模型文件中没有modeling_yi.py啊 HOT 3
- Yi
- Yi 支持LongLoRA吗？ HOT 1
- RuntimeError: You can't move a model that has some modules offloaded to cpu or disk. HOT 3
- Yi-34B-Chat-4bit 无法配合langchain作为agent使用 HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from yi.