Comments (4)
Side note: it'd be good to update the transformers
dependency to the latest (v4.0.0).
from trl.
You are right, when I have time I'll upgrade it to v4.0.0. I haven't tested it but I suspect if you take a model with a text generation head it should work. Note that you need add a value head to your model architecture (see here).
from trl.
I can try it. Other than running with no errors, what other ways I can test that the code is working fine? Is there a benchmark or a quantitative way of verifying the code?
from trl.
Monitoring the rewards on the IMDb dataset would be a good start. For GPT-2 it takes only 1-2h to train.
from trl.
Related Issues (20)
- Minor KTO mis-estimation of KL term on odd batch sizes
- Redundant special tokens when applying chat template (?)
- DataCollatorForCompletionOnlyLM not working with MPS HOT 3
- How to dpo llava?
- How to Add PEFT to PPO Trainer or PPO Config
- Error in AutoModelForCausalLMWithValueHead.load_pretrained("path/to/my/ppo_model")
- Adding Space caused " This instance will be ignored in loss calculation. Note, if this happens often, ..." HOT 1
- losses = losses * self.args.rpo_alpha + policy_nll_loss TypeError: only integer tensors of a single element can be converted to an index HOT 1
- how to use kto_pair loss in the latest version ? HOT 2
- Additional Optional Loss to Center Reward Models HOT 4
- How to Save the PPOTrainer? HOT 3
- Last layer of the Llava-1.5 visual tower is not training
- Is there a way to finetune a finetuned model? HOT 1
- save error HOT 4
- Unexpected Keyword Argument 'add_special_tokens' in LlavaProcessor with LLAVA-1.5-7B HOT 3
- PPOTrainer + AutoModelForCausalLMWithValueHead + Gemma 2 2B -> `RuntimeError: probability tensor contains either `inf`, `nan` or element < 0` HOT 6
- A bug in `__init__` method of class `DPOTrainer`? HOT 4
- Epoch number not right HOT 1
- Error in PPOv2
- How to fine-tune LLaVA using PPO HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from trl.