Comments (1)
Qwen1.5 原版 1200MB 的权重文件中,将 input embeddings 和 output embeddings 分开保存,但实质上它们的值完全相同,我们在保存时会只保留一份 embeddings,因此 Qwen1.5-0.5B 在 16-bit 精度实际上的大小是 900MB,32-bit 精度是 1800MB
在 full 微调时我们保存 32-bit 精度模型(1800MB),lora 微调时我们保存 16-bit 精度模型(900MB)
from llama-factory.
Related Issues (20)
- 如何在训练中指定eval_steps定期评估而不是训练后评估 HOT 1
- LoRA之后合并导出的模型文件为.bin文件而不是safetensors文件 HOT 2
- 使用llava-hf/llava-v1.6-mistral-7b-hf微调报错 HOT 1
- Qwen推理乱回答问题 HOT 3
- 模型微调的并行问题 HOT 1
- 用的什么库实现lora微调 HOT 1
- docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].
- RuntimeError: expected mat1 and mat2 to have the same dtype, but got: float != c10::BFloat16 HOT 1
- 最新代码评估结果generated_predictions.jsonl的顺序乱了,前天还是正常的 HOT 1
- Llama3-70B 8K全参数微调需要多少张A100 80G的卡? HOT 1
- llama factory 训练完后的模型,可以用 huggingface的 transformer 导入吗? HOT 2
- Feature suggestion: cutoff_len could optionally drop too long examples from dataset. HOT 1
- max_grad_norm 在sft下面不起作用
- 用openai库 请求时,流式请求时缺stream_options={"include_usage": True}的处理,用于计算流式tokens
- generation_config.json
- element 0 of tensors does not require grad and does not have a grad_fn HOT 8
- Data shuffle problem HOT 1
- cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_'
- Using lora can not overwrite the Qwen32b's identity, why? HOT 3
- 单机多卡全参数训练LLAMA3,报错`warmup_steps must be either 0 or > 1` HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from llama-factory.