akariasai / attempt Goto Github PK

the json file (attempt/configs/attempt/eval_glue.json) may forget to add the dataset stsb and cola in the paper？And the dataset wnli which is not in the paper and I think that the wnli dataset should be deleted

eval_glue.json:
"test_dataset_name": ["mnli", "qqp", "qnli", "sst2", "rte", "mrpc", "wnli"],

Hoping for your response,
Thanks

Thank you for the greak work! I wonder if you can provide the exact numbers in different model sizes of each method in Figure 4? Looking forward to hearing from you soon.

Hi, thanks for your great works. Could you help to share the code and models? I would like to reproduce the experiments. Many thanks!

Hi! When will you open source all your code and resources?

Can you send me the code and data? I want to have a deeper understanding of your work. My email is [email protected]. Thanks!

Hi
How can I train the model when I provide multiple datasets? It concatenates the datasets, but I know that each batch must have one task. How do you shuffle such batches?

Currently I get error on this line:

    def check_uniqueness(self, samples):
        assert len(np.unique(samples)) == 1

Thanks for sharing your wonderful work!

I wonder about the few-shot learning setting of ATTEMPT.
Did you run a limited number of gradient descent for each datapoint (e.g. 10 GD steps for each shot), or run many GD steps and picked the best score? Also, did you put all the shots in one batch (e.g. 32 shots in a batch = batch-size of 32)?

And also, I wonder about the meaning of k-shot.
Does "k-shot" mean we are provided with k sampled for each label? (e.g. if there are 2 labels like boolq, in total we're given 4x2 = 8 samples?)

Thank you very much.