How instruction_difficulty feature is obtained about alpaca_eval HOT 1 CLOSED

Updated the notebook

First, let's load the instruction complexity ($\mathbf{w'}_x^T$).
This was precomputed by fitting the logistic regression as the one above with the following changes:

• replace the instruction embedding by a one hot encoding of the instruction
• tie the weights of the instruction across all the models and fit jointly across all the models

Here's the equation:

$$win_rate(m,b) = \frac{1}{N} \sum_{i=1}^{N} logistic( \mathbf{w}_l[(m,b)] * tanh(standardized(length(m(x_i)) - length(b(x_i)))) + \mathbf{w}_x*I(x_i) + (\mathbf{w}_m[m] - \mathbf{w}_m[b]))$$

Here $\mathbf{w}_x$ is shared across all models and quantifies how good the baseline win-rate on a certain instruction is.
We then extract this weight and use it as $embedding(x)$ because we want to fit all models disjointly.

Also the paper has more explanation about why: https://arxiv.org/abs/2404.04475

from alpaca_eval.