Comments (3)
Thanks so much!!!
from laser.
That is correct. We do pick LASER hyperparameters for each task and this is important for seeing the huge gains we report. There is an alternate method called LaserRMT that is not from us, which provides a different task-agnostic way to select hyperparameters. I haven't tried it myself but the authors have reported some results.
The simplest way to try LASER across a range of tasks, is to compute a meta-score on a task like AGIEval, and then use it to select the hyperparameter. I am optimistic that we will still see gains across a range of tasks since we find that typically the gains all come from doing intervention in the later MLP layers, and so the optimal hyperparameters tend to have some pattern. The gains might be more modest, compared to only focusing on a single task though.
For most experiments in our paper, we apply LASER to a single layer and in fact we apply a single LASER intervention, i.e., we only edit a single matrix. We have an experiment on GPTJ+CounterFact where we composed multiple LASER interventions. See the paragraph Composing reductions across layers
in the paper. @pratyushasharma has released a script here with details for this experiment, and the upcoming refactoring will support composing LASER in a proper generalizable way.
from laser.
Related to #19
from laser.
Related Issues (19)
- Excellent work, looking forward to following up with further research! HOT 3
- What is the ETA on the code HOT 2
- License HOT 6
- Mistral Support HOT 16
- Where to Get the Dataset HOT 5
- Question HOT 2
- Do you think it could work for MoE models like Mixtral? HOT 2
- Rank-reduced models? HOT 4
- Feature Request for Upcoming Refactoring
- Rank reduction using random matrix theory HOT 1
- what does the 'rate' parameters actually mean in code? HOT 2
- Potential improvements for evaluation HOT 1
- Application to three-dimensional tensors HOT 1
- Llama2-7B + TruthfulQA reproduce issue HOT 8
- method of composing reductions across layers HOT 2
- how to get base model accuracy HOT 2
- Problem Encountered During Reproduction HOT 2
- How to reproduce Figure 5 analysis in this paper?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from laser.