Excellent work, looking forward to following up with further research! about laser HOT 3 CLOSED

Looking forward to the author's reply!

from laser.

Thanks a lot for your comment.

1. Evaluating the other metrics is on our TODO list. I think it shouldnt be hard to compute them and we certainly plan to do so.

2. Your point about finding LASER hyperparameters that work across a range of tasks is also very relevant. One question would be, what model selection criterion does one use? We can take 20% of each datasets and use that as a validation set and compute an aggregate metric. But what metric would make most sense? One can compute average accuracy but the scaling factors and difficulty might be different. Or, one can compute a score that measures the number of datasets we outperform.

   We did notice a common trend which is that typically one needs to do LASER on later MLP layers with a significant amount of reduction to get the best performance. This is mostly but not always true for the results in Table 1 (see selected hyperparameters in Table 3). I recently tested this hypothesis for the Phi-1.5 LLM on the Counterfact dataset. The base model gives an 10% accuracy and I tried just two choices for LASER: the last transformer layer, keeping just 0.01 of the original rank, and using the first or the second layer of the MLP. This choice is guided by the above pattern in Table 3. And this already gave a performance boost of 16%. Therefore, I am somewhat optimistic that we can find a general choice for LASER that outperforms the base model in the majority of settings. The code for running the Phi-1.5 experiment is here: https://github.com/pratyushasharma/laser/blob/main/src/intervention_phi15_counterfact.py

Thanks a lot for your comments and the reference to the Meng et al paper which we cited and which we really like. Please let us know if we can help you in using this codebase.

from laser.

Closing this issue. Please feel free to reopen it up, or send us an email. We will update the README.md once we have more results (adding @pratyushasharma to this thread as well).

from laser.