Comments (8)
多卡需要每张卡都有 24G 内存。
from llama-factory.
多卡需要每张卡都有 24G 内存。
Dalao,单卡我这里也有个问题,13b的模型我4bit量化,输入512输出512怎么微调也OOM o(╥﹏╥)o
from llama-factory.
GPU 的空闲显存有多少?
从 512 减少到 256 试试呢?
from llama-factory.
GPU 的空闲显存有多少? 从 512 减少到 256 试试呢?
降低了可以。还有个问题
from transformers import LlamaForCausalLM, AutoTokenizer
import torch
ckpt = './bloom_13b/'
device = torch.device('cuda')
model = LlamaForCausalLM.from_pretrained(ckpt, device_map={"":0},load_in_8bit=True, low_cpu_mem_usage=True)
tokenizer = AutoTokenizer.from_pretrained(ckpt)
model.eval()
prompt = "XXXXXXXXXXXXXXXXXXXXXXXXXXX"
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(device)
generate_ids = model.generate(input_ids, max_new_tokens=500, do_sample = False, repetition_penalty=1., eos_token_id=2, bos_token_id=1, pad_token_id=0)
output = tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
response = output[len(prompt):]
print(response)
官方的示例对话的结果,对比我用项目里的web demo结果不一致(均为8bit do_sample=False),而且web生成的效果比我用上述官方示例的效果差很多,这是怎么回事?
web demo的参数我在后台都改了,如下
gen_kwargs = {
"input_ids": input_ids,
"do_sample": False,
"top_p": 0.01,
"temperature": 0.99,
"num_beams": 1,
# "max_length": max_length,
"max_new_tokens":500,
"repetition_penalty": 1.0,
"logits_processor": get_logits_processor(),
"streamer": streamer,
"eos_token_id":2,
"bos_token_id":1,
"pad_token_id":0
}
from llama-factory.
web demo的回答惜字如金。。
from llama-factory.
web demo的回答惜字如金。。
已经解决,原来在代码里面,会自动包装问题,导致结果和官方示例不一致,而且还导致回答惜字如金
from llama-factory.
web demo的回答惜字如金。。
已经解决,原来在代码里面,会自动包装问题,导致结果和官方示例不一致,而且还导致回答惜字如金
你好,请问怎么修改解决保证该项目与官方回答基本一致?
from llama-factory.
@yumulinfeng1 使用指令微调后的模型时候应该在命令行参数中加入 --prompt_template
参数
from llama-factory.
Related Issues (20)
- 用最新的代码全量微调llama-3-70B报错 HOT 2
- 我跑dpo的时候遇到了ZeroDivisionError: integer division or modulo by zero的error HOT 1
- 训练一段时间后,在保存文件时,会提示文件夹【拒绝访问】 HOT 1
- 单机多卡,streaming 下,开始迭代数据内存会暴涨,随着迭代后持续慢慢涨到爆;非streaming 模式正常; HOT 2
- LLaMA3-8B won't inference HOT 6
- HELP:能否支持讯飞星火开源13B模型的训练? HOT 1
- [Help]: Dataset PPO empty answer field HOT 4
- error when using XuanYuan-70B-int4-Chat HOT 1
- Component' is not defined HOT 2
- Megatron支持 HOT 1
- Llama30-70B-instruct lora 微调 无法读入tokenzier 错误信息为如下图片中信息 HOT 1
- Failed to Reproduce Results as Described in 4.2 HOT 5
- Two issues with potential model format and extended training length HOT 1
- dpo训练验证问题 HOT 1
- 以lora方式SFT微调之后,使用beam search推理时模型输出有时候不能正常停止
- 这个shuffle是不是应该写在前面 HOT 1
- ValueError: Current model is not supported by mixture-of-depth. HOT 1
- Baichuan2权重合并问题 HOT 4
- belle_multiturn.py文件处理数据集后的数据可以放一下么?
- web推理Llama3-8B-Chinese-Chat,出现AttributeError: 'NoneType' object has no attribute 'get' HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from llama-factory.