timoschick / pet Goto Github PK

Thanks for your work!
I see most experiments in the paper "Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference" used RoBERTa large model.
So, can the BERT model get similar results with RoBERTa model?
Did you try the BERT model?

Hi
As I train the model, I see some outputs like below, could you kindly comment how I can visualize the eval accuracies and remove these outputs? thanks

2021-08-21 14:48:28,778 - INFO - wrapper - --- Example 4 ---
2021-08-21 14:48:28,779 - INFO - wrapper - input_ids         = ['<s>', 'Ġ"', 'ĠSecurity', 'Ġforces', 'Ġwere', 'Ġon', 'Ġhigh', 'Ġalert', 'Ġafter', 'Ġa', 'Ġcampaign', 'Ġmar', 'red', 'Ġby', 'Ġviolence', 'Ġ"', 'Ġ?', '</s>', '</s>', '<mask>', 'Ġ,', 'Ġ"', 'ĠSecurity', 'Ġforces', 'Ġwere', 'Ġon', 'Ġhigh', 'Ġalert', 'Ġafter', 'Ġan', 'Ġelection', 'Ġcampaign', 'Ġin', 'Ġwhich', 'Ġmore', 'Ġthan', 'Ġ1', ',', '000', 'Ġpeople', ',', 'Ġincluding', 'Ġseven', 'Ġelection', 'Ġcandidates', ',', 'Ġhave', 'Ġbeen', 'Ġkilled', '.', 'Ġ"', '</s>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>', '<pad>']
attention_mask    = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
token_type_ids    = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
mlm_labels        = [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]
logits            = [-1]
label             = 0

/pet/tasks.py should include import log

Gives key error in labels, as in the MNLI train dataset, there are no labels.

Line 41, pet/tasks.py
function _create_examples in MnliProcessor is where it creates examples, and assigns labels as line[-1], which is an empty string for training data of MNLI dataset,
In MNLI training dataset, each line ends with:
....sentence b. ', '31193', '31193n', 'government', 'neutral', '', '', '', '']
those 4 empty strings at the end

2021-05-18 08:52:37,308 - INFO - wrapper - Writing example 0
Traceback (most recent call last):
File "cli.py", line 282, in
main()
File "cli.py", line 263, in main
no_distillation=args.no_distillation, seed=args.seed)
File "/content/gdrive/My Drive/pet/pet/pet/modeling.py", line 249, in train_pet
save_unlabeled_logits=not no_distillation, seed=seed)
File "/content/gdrive/My Drive/pet/pet/pet/modeling.py", line 355, in train_pet_ensemble
unlabeled_data=unlabeled_data))
File "/content/gdrive/My Drive/pet/pet/pet/modeling.py", line 434, in train_single_model
results_dict['train_set_before_training'] = evaluate(model, train_data, eval_config)['scores']['acc']
File "/content/gdrive/My Drive/pet/pet/pet/modeling.py", line 490, in evaluate
n_gpu=config.n_gpu, decoding_strategy=config.decoding_strategy, priming=config.priming)
File "/content/gdrive/My Drive/pet/pet/pet/wrapper.py", line 354, in eval
eval_dataset = self._generate_dataset(eval_data, priming=priming)
File "/content/gdrive/My Drive/pet/pet/pet/wrapper.py", line 401, in _generate_dataset
features = self._convert_examples_to_features(data, labelled=labelled, priming=priming)
File "/content/gdrive/My Drive/pet/pet/pet/wrapper.py", line 426, in _convert_examples_to_features
input_features = self.preprocessor.get_input_features(example, labelled=labelled, priming=priming)
File "/content/gdrive/My Drive/pet/pet/pet/preprocessor.py", line 83, in get_input_features
label = self.label_map[example.label] if example.label is not None else -100
KeyError: ''

If you follow the traceback, you can find that the code is trying to run eval on the train set, which is unlabeled as I mentioned above.

How to train on MNLI dataset.

Hi. I want to train PET on a new task for which I prepared custom_task_processor.py and custom_task_pvp.py. My question is how should we run/tell the program to read our customized files (instead of the main files) and run the registered new task? It seems that just running the commands under the PET Training and Evaluation section does not do the task.

How should I run codes for generation tasks such as cnn-dailymail?

I would love to see this in the huggingace transformers repo, do you have any plans about this ?
There is already an open issue huggingface/transformers#7370

Dear @timoschick

I have read and found you paper "Few-Shot Text Generation with Pattern-Exploiting Training" really interesting.
Can you share with us the code you use to conduct experiments in your paper?

Assumed to have a spam/ham dataset (like the simple SMS Spam dataset like in the following:

[ham]
Ok lar... Joking wif u oni...

[spam]
Free entry in 2 a wkly comp to win FA Cup final tkts 21st May 2005. Text FA to 87121 to receive entr...

How would you define a pattern for Pi(a), for s given input text a and the resulting verbalizer v?

Another case, is a dataset like the Hate Speech Dataset (A Measurement Study of Hate Speech in Social Media)

6,"i hate asian people, especially the filipinos and pakis. ",1402091070021,"Confessions,japan,tourist destinations,Travel",Canada,Manitoba,asian people,ethnicity
9,I can't stand white people. They make me furious.. And I'm white,1402409977473,"Confessions,racism,society",United States,California,white people,race

where we have the following labels: behavior, race, sexual_orientation, other, ethnicity, physical, class, religion, defined, and the sentences have been selected looking at the following patterns:

So how the define the patterns Pi? My guess is that it one could be something like

P1(a) = I...
P2(a) = I cant'...
P3(a) = I don't like...
...

where

v(1) = hate
v(2) = stand
v(3) = like
...

Does it make sense? In that case, how a dataset row would look like?
Thanks a lot!

For example, I am trying task BoolQ, when I get the dataset from SuperGLUE website and unzip the file. I get train.jsonl , val.jsonl and test.jsonl.But from your code, I must get another file unlabeled.jsonl. So, what is that and how to get it?

Here are the commands I am running to install GenPET:

git clone --single-branch --branch feature/genpet https://github.com/timoschick/pet
pip install -r pet/requirements.txt

Then, when I run this:

python pet/cli.py --help

It throws an unexpected error:

Traceback (most recent call last):
  File "pet/cli.py", line 328, in <module>
    main()
  File "pet/cli.py", line 259, in main
    args = parser.parse_args()
  File "/usr/lib/python3.7/argparse.py", line 1764, in parse_args
    args, argv = self.parse_known_args(args, namespace)
  File "/usr/lib/python3.7/argparse.py", line 1796, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
  File "/usr/lib/python3.7/argparse.py", line 2002, in _parse_known_args
    start_index = consume_optional(start_index)
  File "/usr/lib/python3.7/argparse.py", line 1942, in consume_optional
    take_action(action, args, option_string)
  File "/usr/lib/python3.7/argparse.py", line 1870, in take_action
    action(self, namespace, argument_values, option_string)
  File "/usr/lib/python3.7/argparse.py", line 1052, in __call__
    parser.print_help()
  File "/usr/lib/python3.7/argparse.py", line 2490, in print_help
    self._print_message(self.format_help(), file)
  File "/usr/lib/python3.7/argparse.py", line 2474, in format_help
    return formatter.format_help()
  File "/usr/lib/python3.7/argparse.py", line 293, in format_help
    help = self._root_section.format_help()
  File "/usr/lib/python3.7/argparse.py", line 224, in format_help
    item_help = join([func(*args) for func, args in self.items])
  File "/usr/lib/python3.7/argparse.py", line 224, in <listcomp>
    item_help = join([func(*args) for func, args in self.items])
  File "/usr/lib/python3.7/argparse.py", line 224, in format_help
    item_help = join([func(*args) for func, args in self.items])
  File "/usr/lib/python3.7/argparse.py", line 224, in <listcomp>
    item_help = join([func(*args) for func, args in self.items])
  File "/usr/lib/python3.7/argparse.py", line 540, in _format_action
    help_text = self._expand_help(action)
  File "/usr/lib/python3.7/argparse.py", line 629, in _expand_help
    return self._get_help_string(action) % params
TypeError: %o format: an integer is required, not dict

Would you mind sharing exactly what to do to download the training data and do training and evaluation for one example? I'm having trouble figuring out exactly how things should be arranged and what to put for the command line parameters. Any example would do, as long as I can just follow the instructions.

Hi,

I try to reproduce your result of roberta-base on MNLI. Your result listed in paper is 55.1, while I only get 33.7.

I run the code with
python3 cli.py
--method pet
--pattern_ids 0 1 2 3
--data_dir $DATA_DIR
--model_type roberta
--model_name_or_path roberta-base
--task_name mnli
--output_dir result/mnli
--lm_training
--alpha 1e-4
--pet_per_gpu_train_batch_size 1
--pet_per_gpu_unlabeled_batch_size 3
--pet_max_seq_length 256
--pet_gradient_accumulation_steps 4
--pet_max_steps 1000
--learning_rate 1e-5
--sc_max_seq_length 256
--sc_per_gpu_train_batch_size 4
--sc_gradient_accumulation_steps 4
--sc_num_train_epochs 3
--train_examples 50
--unlabeled_examples 30000
--split_examples_evenly
--do_train
--do_eval
which strictly follow the setting mentioned in your paper. Could you please tell me how to fix it?

Regards,
Matthew

Thank you for releasing your codes.

I have found a small typo in your EACL2021 paper.

In the explanation for MNLI in Sec 4.1, the followings should be opposite in the verbalizers examples.

• "Wrong" and "Right"
• "No" and "Yes"

Thanks.

I am not sure if this is already found in the code for the identity function solution in your recent paper:

When I want to use multi-token labels - for example:

1. very good
2. good
3. ok
4. bad
   5 very bad

very good / very bad are multi-token.

Can you give me a short (for dummys) idea how to implement train that?

Amazing work!

I wanted to check regarding an error, while trying to label unlabeled data it throws an error.
my unlabeled data CSV is of the form text_a , text_b , "," not sure what could be a the issue here.

Thanks for sharing this repo. When looking at the /examples dir, you split your dataset (labeled data?) to

• DEV_FILE_NAME
• TRAIN_FILE_NAME
• TEST_FILE_NAME

& further

• UNLABELED_FILE_NAME

Two questions arise:
a) How do you split the labeled data (distribution, e.g. are you splitting 32 training examples from fewglue to DEV / TRAIN / TEST equally ?)
b) will UNLABELED be automatically predicted and how is the result stored

gpt2 is missing the SEQUENCE_CLASSIFIER_WRAPPER; GPT2ForSequenceClassification, which is available in the transformer module.

In what format should the csv files listed in the taskprocessor be?
What should the row names in the csv files be?
Examples of files like this?

I know PET is only for fine tuning pre-trained models, but is it possible to fully train a model of your own with PET? From scratch, like?

The paper said GenPET wouldn't have a verbalizer. I know you said that the code in that feature branch had some stuff that wasn't talked about in the arxiv preprint. Is this one of them?

I would like to use pet for imbalanced data (Class1 ~80% of data, Class 2-5 ~20% of data)
Do you have experience with or training tips for using PET with imbalanced classes?

Hi. Thanks for the great repo. I have got a question regarding the PET training and annotating an unlabeled set (as mentioned in the paper examples from D). I assume that it would be done using the command in the PET Training and Evaluation section in the repo. However, I am not sure where to put the unlabeled set and where to get the predicted labels? Would you please let me know how we should get the predicted labels for the unlabeled set? Thank you.

If max_steps or data length is not divisible by gradient_accumulation_steps some gradients are lost. Since updating only takes place at if (step + 1) % gradient_accumulation_steps == 0:

Hi Timo Schick,

Thanks for creating this repo. I've been looking to implement this using TensorFlow 2.4 and multi-gpu setup for dialog-nlu task. Are you planning to release Tensorflow 2.4 version for the same which can be converted to tflite to test in mobile devices.

Thanks.

Hi!
This model can be used for romanian language?

I am training a pet model on 500gb of text. I have properly processed the data, but I can't load all my data into a variable since I don't have nearly enough RAM to do that.

Hi, do you have any instructions on how to train a petal model and replicate the petal results?

Thanks,
Yulong

Hi guys,
PET is a great concept that you have introduced and I am really excited to be working on it. I am however facing an error while running a personalized task -

python cli.py --method pet --pattern_ids 0 --data_dir D:\personal_projects --model_type albert --model_name_or_path albert-xxlarge-v2 --task_name pycode-generation --output_dir D:\personal_projects\output --do_train --do_eval 2020-12-17 17:22:45,587 - INFO - cli - Parameters: Namespace(adam_epsilon=1e-08, alpha=0.9999, cache_dir='', data_dir='D:\\personal_projects', decoding_strategy='default', do_eval=True, do_train=True, eval_set='dev', ipet_generations=3, ipet_logits_percentage=0.25, ipet_n_most_likely=-1, ipet_scale_factor=5, learning_rate=1e-05, lm_training=False, logging_steps=50, max_grad_norm=1.0, method='pet', model_name_or_path='albert-xxlarge-v2', model_type='albert', no_cuda=False, no_distillation=False, output_dir='D:\\personal_projects\\output', overwrite_output_dir=False, pattern_ids=[0], pet_gradient_accumulation_steps=1, pet_max_seq_length=256, pet_max_steps=-1, pet_num_train_epochs=3, pet_per_gpu_eval_batch_size=8, pet_per_gpu_train_batch_size=4, pet_per_gpu_unlabeled_batch_size=4, pet_repetitions=3, priming=False, reduction='wmean', sc_gradient_accumulation_steps=1, sc_max_seq_length=256, sc_max_steps=-1, sc_num_train_epochs=3, sc_per_gpu_eval_batch_size=8, sc_per_gpu_train_batch_size=4, sc_per_gpu_unlabeled_batch_size=4, sc_repetitions=1, seed=42, split_examples_evenly=False, task_name='pycode-generation', temperature=2, test_examples=-1, train_examples=-1, unlabeled_examples=-1, verbalizer_file=None, warmup_steps=0, weight_decay=0.01, wrapper_type='mlm') 2020-12-17 17:22:45,590 - INFO - tasks - Creating features from dataset file at D:\personal_projects (num_examples=-1, set_type=train) 2020-12-17 17:22:45,597 - INFO - tasks - Returning 26 train examples with label dist.: [('code', 1), ('1', 7), ('2', 7), ('3', 5), ('4', 6)] 2020-12-17 17:22:45,597 - INFO - tasks - Creating features from dataset file at D:\personal_projects (num_examples=-1, set_type=dev) 2020-12-17 17:22:45,600 - INFO - tasks - Returning 16 dev examples with label dist.: [('code', 1), ('1', 4), ('3', 3), ('2', 4), ('4', 4)] 2020-12-17 17:22:45,601 - INFO - tasks - Creating features from dataset file at D:\personal_projects (num_examples=-1, set_type=unlabeled) 2020-12-17 17:22:45,605 - INFO - tasks - Returning 6 unlabeled examples with label dist.: [('1', 6)] 2020-12-17 17:22:55,331 - INFO - wrapper - Writing example 0 Traceback (most recent call last): File "cli.py", line 455, in <module> main() File "cli.py", line 436, in main no_distillation=args.no_distillation, seed=args.seed) File "D:\personal_projects\pet\modeling.py", line 249, in train_pet save_unlabeled_logits=not no_distillation, seed=seed) File "D:\personal_projects\pet\modeling.py", line 355, in train_pet_ensemble unlabeled_data=unlabeled_data)) File "D:\personal_projects\pet\modeling.py", line 434, in train_single_model results_dict['train_set_before_training'] = evaluate(model, train_data, eval_config)['scores']['acc'] File "D:\personal_projects\pet\modeling.py", line 490, in evaluate n_gpu=config.n_gpu, decoding_strategy=config.decoding_strategy, priming=config.priming) File "D:\personal_projects\pet\wrapper.py", line 352, in eval eval_dataset = self._generate_dataset(eval_data, priming=priming) File "D:\personal_projects\pet\wrapper.py", line 399, in _generate_dataset features = self._convert_examples_to_features(data, labelled=labelled, priming=priming) File "D:\personal_projects\pet\wrapper.py", line 424, in _convert_examples_to_features input_features = self.preprocessor.get_input_features(example, labelled=labelled, priming=priming) File "D:\personal_projects\pet\preprocessor.py", line 83, in get_input_features label = self.label_map[example.label] if example.label is not None else -100 KeyError: 'code'

Could you explain me on why am I getting this key error ?

Having get_dev_examples() method to return the same data as the get_unlabeled_examples() method
raised this error. It seems that it expects labels for dev.csv. However, the unlabeled.csv does not have labels and has , instead.

File "/pet-master/pet/preprocessor.py", line 83, in get_input_features label = self.label_map[example.label] if example.label is not None else -100 KeyError: ','

Thanks for the great repo. I have got 2 questions.

1. How many days you expect a customized task with sizes of the train: 610, dev: 500, test: 1000, unlabeled data: 2000 would take on a CPU machine? It's been four days and the code is still going :D this is the current step "learning_rate": 3.1000000000000004e-06, "loss": 0.007363705812799708, "step": 3450}

2. To get the predictions/final_labels for unlabeled data which folder/file should we look at?
   I have 7 folders in my output_directory so far. p0-i0, p0-i1, p0-i2, p1-i0, p1-i0, p1-i1, p1-i2, final I designed 2 patterns (0,1) for my customized_PVP.
   There are two files in my output_directory as well: result_test.txt and unlabeled_logits.txt. The first three lines in my unlabeled_logits.txt:
   -1
   8.052350375527427 5.843138097046414
   14.612752109704642 2.8329479116033762

Hello! I'm trying to replicate your work, and I'm currently comparing the performance of my replication to your implementation. Just to be sure, could you please provide me the exact commands you used for training the SuperGLUE tasks? I would be very grateful. Thank you!

What should be there in the Unlabeled JSON file?
https://github.com/timoschick/pet/blob/master/pet/tasks.py#L630

Hi,
Congratulations on this great work.
I have a question about how to use PET for the multi-label classification task. Do I need to change the loss function defined for the current training?

Hello, @timoschick.
First of all, I want to compliment you for the great work you did with PET.
I think this is amazing and I can't stand the idea to try it to solve some of my data problems. I am quite new to transformers, so probably I'm doing something terribly naive.
First of all I created my personalized task using
`class MyTaskDataProcessor(DataProcessor):
"""
Example for a data processor.
"""

# Set this to the name of the task
TASK_NAME = "illlegal-detection"

# Set this to the name of the file containing the train examples
TRAIN_FILE_NAME = "train.csv"

# Set this to the name of the file containing the dev examples
DEV_FILE_NAME = "dev.csv"

# Set this to the name of the file containing the test examples
TEST_FILE_NAME = "test.csv"

# Set this to the name of the file containing the unlabeled examples
UNLABELED_FILE_NAME = "unlabeled.csv"

# Set this to a list of all labels in the train + test data
LABELS = ["0", "1"]

# Set this to the column of the train/test csv files containing the input's text a
TEXT_A_COLUMN = 0

# Set this to the column of the train/test csv files containing the input's text b or to -1 if there is no text b
TEXT_B_COLUMN = -1

# Set this to the column of the train/test csv files containing the input's gold label
LABEL_COLUMN = 1

def get_train_examples(self, data_dir: str) -> List[InputExample]:
    """
    This method loads train examples from a file with name `TRAIN_FILE_NAME` in the given directory.
    :param data_dir: the directory in which the training data can be found
    :return: a list of train examples
    """
    return self._create_examples(os.path.join(data_dir, MyTaskDataProcessor.TRAIN_FILE_NAME), "train")

def get_dev_examples(self, data_dir: str) -> List[InputExample]:
    """
    This method loads dev examples from a file with name `DEV_FILE_NAME` in the given directory.
    :param data_dir: the directory in which the dev data can be found
    :return: a list of dev examples
    """
    return self._create_examples(os.path.join(data_dir, MyTaskDataProcessor.DEV_FILE_NAME), "dev")

def get_test_examples(self, data_dir) -> List[InputExample]:
    """
    This method loads test examples from a file with name `TEST_FILE_NAME` in the given directory.
    :param data_dir: the directory in which the test data can be found
    :return: a list of test examples
    """
    return self._create_examples(os.path.join(data_dir, MyTaskDataProcessor.TEST_FILE_NAME), "test")

def get_unlabeled_examples(self, data_dir) -> List[InputExample]:
    """
    This method loads unlabeled examples from a file with name `UNLABELED_FILE_NAME` in the given directory.
    :param data_dir: the directory in which the unlabeled data can be found
    :return: a list of unlabeled examples
    """
    return self._create_examples(os.path.join(data_dir, MyTaskDataProcessor.UNLABELED_FILE_NAME), "unlabeled")

def get_labels(self) -> List[str]:
    """This method returns all possible labels for the task."""
    return MyTaskDataProcessor.LABELS

def _create_examples(self, path, set_type, max_examples=-1, skip_first=0):
    """Creates examples for the training and dev sets."""
    examples = []

    with open(path) as f:
        reader = csv.reader(f, delimiter=',')
        for idx, row in enumerate(reader):
            guid = "%s-%s" % (set_type, idx)
            label = row[MyTaskDataProcessor.LABEL_COLUMN]
            text_a = row[MyTaskDataProcessor.TEXT_A_COLUMN]
            text_b = row[MyTaskDataProcessor.TEXT_B_COLUMN] if MyTaskDataProcessor.TEXT_B_COLUMN >= 0 else None
            example = InputExample(guid=guid, text_a=text_a, text_b=text_b, label=label)
            examples.append(example)

    return examples

PROCESSORS[MyTaskDataProcessor.TASK_NAME] = MyTaskDataProcessor`

Ok, now I want to use it to solve my problem. So I'm using this command line:
cmd = """python cli.py --method pet --data_dir .../comments_class/code/pet-master/pet --model_type roberta --model_name_or_path roberta --task_name --output_dir ...comments_class/data/output -- pattern_ids 0 1 --do_train --do_eval """
The problem is: how to communicate to the cli.py the new task I've created?
Sorry if I'm being too naive. I think this is an easy one and maybe it will be useful for future noobs too.
Thank again for your work!

Hi, I am trying to reproduce the result of the RTE task. I followed the method in #4 (comment) and get the following output in my terminal.

2021-08-28 14:39:43,537 - INFO - modeling - --- RESULT (pattern_id=0, iteration=0) ---
2021-08-28 14:39:43,537 - INFO - modeling - {'acc': 0.7075812274368231}
2021-08-28 14:39:43,557 - INFO - modeling - === OVERALL RESULTS ===
2021-08-28 14:39:43,558 - INFO - modeling - acc-p0: 0.7075812274368231 +- 0
2021-08-28 14:39:43,558 - INFO - modeling - acc-all-p: 0.7075812274368231 +- 0

Since I used the --do_eval argument, the result should be that on dev set, but the result shown in Table1 in your paper is 69.8 (mine is 70.8). Did I use the correct training arguments? And here is my command for training

python cli.py \
--method pet \
--pattern_ids 0 1 2 3 \
--data_dir $PET/dataset/RTE \
--model_type albert \
--model_name_or_path $PETmodels/albert-xxlarge-v2 \
--task_name rte \
--output_dir ./rte-output-large-serious-2 \
--lm_training \
--pet_per_gpu_train_batch_size 2 \
--pet_gradient_accumulation_steps 8 \
--pet_max_steps 250 \
--sc_per_gpu_unlabeled_batch_size 2 \
--sc_gradient_accumulation_steps 8 \
--do_train \
--do_eval \
--sc_max_steps 5000

If I want to reproduce the results of the other 7 taks in Table1 in your paper, can I simply change the --task_name argument?

THANKS IN ADVANCE!

fresh dir, syncd the repo, did the pip install from requirements.txt

command line

python3 cli.py --method pet --pattern_ids 0 1 2 3 --data_dir MNLI/ --model_type roberta --model_name_or_path roberta-large --task_name mnli --output_dir out2 --do_train --do_eval

Error:

Evaluating:  20%|█████████████▍                                                     | 9820/49088 [24:35<1:38:18,  6.66it/s]
Traceback (most recent call last):
  File "cli.py", line 282, in <module>
    main()
  File "cli.py", line 263, in main
    no_distillation=args.no_distillation, seed=args.seed)
  File "/home/qblocks/shan/pet/pet/modeling.py", line 249, in train_pet
    save_unlabeled_logits=not no_distillation, seed=seed)
  File "/home/qblocks/shan/pet/pet/modeling.py", line 355, in train_pet_ensemble
    unlabeled_data=unlabeled_data))
  File "/home/qblocks/shan/pet/pet/modeling.py", line 434, in train_single_model
    results_dict['train_set_before_training'] = evaluate(model, train_data, eval_config)['scores']['acc']
  File "/home/qblocks/shan/pet/pet/modeling.py", line 490, in evaluate
    n_gpu=config.n_gpu, decoding_strategy=config.decoding_strategy, priming=config.priming)
  File "/home/qblocks/shan/pet/pet/wrapper.py", line 376, in eval
    logits = EVALUATION_STEP_FUNCTIONS[self.config.wrapper_type](self)(batch)
  File "/home/qblocks/shan/pet/pet/wrapper.py", line 525, in mlm_eval_step
    return self.preprocessor.pvp.convert_mlm_logits_to_cls_logits(batch['mlm_labels'], outputs[0])
  File "/home/qblocks/shan/pet/pet/pvp.py", line 207, in convert_mlm_logits_to_cls_logits
    masked_logits = logits[mlm_labels >= 0]
RuntimeError: copy_if failed to synchronize: cudaErrorLaunchFailure: unspecified launch failure

Hello!

I would like you kindly to revise this question, if possible:
I am trying to run the project using this setup (f.e: for RTE task) and with all dependencies installed from requirements.txt
(from master branch)

!python cli.py
--method pet      
--model_type albert \
--data_dir data/FewGLUE/RTE \
--task_name rte \
--model_name_or_path albert-xxlarge-v2 \
--output_dir ./models/petModels/rte_albert \
--do_train \
--do_eval \
--pet_per_gpu_eval_batch_size 8 \
--pet_per_gpu_train_batch_size 2 \
--pet_gradient_accumulation_steps 8 \
--pet_max_steps 250 \
--pet_max_seq_length 256 \
--pet_repetitions 3 \
--sc_per_gpu_train_batch_size 2 \
--sc_per_gpu_unlabeled_batch_size 2 \
--sc_gradient_accumulation_steps 8 \
--sc_max_steps 5000 \
--sc_max_seq_length 256 \
--sc_repetitions 1

and I am obtaining the following error only when I am trying to use multiple GPU s (for a machine with a single GPU, it works)

Traceback (most recent call last):
  File "cli.py", line 283, in <module>
    main()
  File "cli.py", line 264, in main
    no_distillation=args.no_distillation, seed=args.seed)
  File "/$PATH1/pet/pet/modeling.py", line 249, in train_pet
    save_unlabeled_logits=not no_distillation, seed=seed)
  File "/$PATH1/pet/pet/modeling.py", line 355, in train_pet_ensemble
    unlabeled_data=unlabeled_data))
  File "/$PATH1/pet/pet/modeling.py", line 459, in train_single_model
    temperature=config.temperature
  File "/$PATH1/pet/pet/wrapper.py", line 300, in train
    loss = TRAIN_STEP_FUNCTIONS[self.config.wrapper_type](self)(batch, **train_step_inputs)
  File "/$PATH1/pet/pet/wrapper.py", line 478, in mlm_train_step
    outputs = self.model(**inputs)
  File "/$PATH2/anaconda3/envs/python3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/$PATH2/anaconda3/envs/python3/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 155, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/$PATH2/anaconda3/envs/python3/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 165, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/$PATH2/anaconda3/envs/python3/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 85, in parallel_apply
    output.reraise()
  File "/$PATH2/anaconda3/envs/python3/lib/python3.6/site-packages/torch/_utils.py", line 395, in reraise
    raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/$PATH2/anaconda3/envs/python3/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
    output = module(*input, **kwargs)
  File "/$PATH2/anaconda3/envs/python3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/$PATH2/anaconda3/envs/python3/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 155, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/$PATH2/anaconda3/envs/python3/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 165, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/$PATH2/anaconda3/envs/python3/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 85, in parallel_apply
    output.reraise()
  File "/$PATH2/anaconda3/envs/python3/lib/python3.6/site-packages/torch/_utils.py", line 395, in reraise
    raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in replica 1 on device 1.
Original Traceback (most recent call last):
  File "/$PATH2/anaconda3/envs/python3/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
    output = module(*input, **kwargs)
  File "/$PATH2/anaconda3/envs/python3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/$PATH2/anaconda3/envs/python3/lib/python3.6/site-packages/transformers/modeling_albert.py", line 814, in forward
    output_hidden_states=output_hidden_states,
  File "/$PATH2/anaconda3/envs/python3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/$PATH2/anaconda3/envs/python3/lib/python3.6/site-packages/transformers/modeling_albert.py", line 556, in forward
    input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
  File "/$PATH2/anaconda3/envs/python3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/$PATH2/anaconda3/envs/python3/lib/python3.6/site-packages/transformers/modeling_bert.py", line 178, in forward
    inputs_embeds = self.word_embeddings(input_ids)
  File "/$PATH2/anaconda3/envs/python3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/$PATH2/anaconda3/envs/python3/lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 114, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/$PATH2/anaconda3/envs/python3/lib/python3.6/site-packages/torch/nn/functional.py", line 1724, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: arguments are located on different GPUs at /pytorch/aten/src/THC/generic/THCTensorIndex.cu:403

Do you have any ideas of what shall I change in order to work, or if am I doing something wrong?
I will apreciate any hints or remarks, if possible!

How do you get the prediction from the final supervised classifier which is there in final folder to get prediction on new sentnence

Hi @timoschick, thanks very much for your work, I have a question about how you decide the unlabelled data for each task.

In the paper you say

Similarly, we construct the set D of unlabeled examples by selecting 10000 examples per label and removing all labels

Taking agnews as an example, I assume it means you take 40,000 examples (it has 4 classes in total) from training in total and there will be 10,000 examples for each class. However, in your code, it seems that you are not following the 10,000 examples per label thing by just shuffling and picking first 40,000 examples.

I am little bit confused about this, any clarification would be helpful.

First of all, thanks for supporting multiple token verbalizers. This feature is critical to my scenario. My verbalizers typically have 4-6 tokens.

As mentioned in https://github.com/timoschick/pet#pet-with-multiple-masks, using multi-mask PET is very very slow at evaluation step because 1) max eval batch size has to be 1; 2) Multiple forward passes needs to be run for each eval step.

My questions are:

1. Is there any plan to address the evaluation step speed issue with some new design?
2. Is there any alternative way to speed up the evaluation step?

Thanks.

I was wondering if the code for https://arxiv.org/pdf/2010.13641.pdf is included in this repository or somewhere else? This is the link in the paper.
Thanks in advance!

It seems that your code will utilize all GPU，but I share a server with others, and it turns out to be something like cuda out of memory. Is there any parameters that can assign which GPU to use?

This thing looks amazing so far! Anyone tried using it for minimized costs GPT training? If yes, can anyone share examples of generated text, and preferably, how to do it?

Hi,

During my PET training, I found it is helpful to tune the last sequence classifier stage individually. What I plan to do is: 1) run PET method as normal; 2) In separate programs, load the merged unlabeled_logits.txt and tune sequence classifier parameters. I'd like to do step2 individually without reruning full PET.

I hit an issue when I tried to do the above step 2. While expected accuracy for my dataset is ~50%, I only got 1.5%. I used the same parameters for sequence classifier as I did for PET.

The following is what I did:

1. change this cli.py line to use_logits=True;
2. add an option in modeling.py train_classifier() so that it will load given logits file (unlabeled_logits.txt) like this line, then assign the logits to unlabeled data.

My questions are: what am I missing? What's the proper way to run sequence classifier with pre-trained/merged logits file?

Thanks.

I trained a model using pet and I want to evaluate it over some validation set. While I remove --do_train and only leave --do_eval option, it still loads training data which in my opinion is unnecessary,

First, thanks for sharing.

I have been trying to make classification models using BERT as base. When since i am trying to use Japanese model, i used cl-tohoku models from hugging face. Training classification using hugging face and custom MLP models for my case achieved around 70% accuracy. I have been trying to improve the performance, then i saw this repository.

Since the classification uses multiple words as a class, i used command like this

python3 cli.py --method pet \
--pattern_ids 0 1 2 3 4 5\
--model_type bert \
--model_name_or_path cl-tohoku/bert-base-japanese-whole-word-masking \
--data_dir ../custom-data \
--task_name custom \
--output_dir custom-output \
--do_train \
--do_eval \
--train_examples 500 \
--unlabeled_examples 500 \
--split_examples_evenly \
--pet_per_gpu_train_batch_size 1 \
--pet_per_gpu_unlabeled_batch_size 1 \
--pet_gradient_accumulation_steps 1 \
--pet_max_steps 250 \
--lm_training \
--sc_per_gpu_train_batch_size 1 \
--sc_per_gpu_unlabeled_batch_size 1 \
--sc_gradient_accumulation_steps 1 \
--sc_max_steps 5000 \
--pet_per_gpu_eval_batch_size 1 \
--sc_max_seq_length 512 \
--pet_max_seq_length 512 \
--pet_repetitions 3

At first, i tried to use train_examples 10, i only got 0.04 accuracy
when i increased train_examples to 500, i got 0.3 accuracy. Is this expected?

I tried to read the paper, this train_examples is used for making PLM right? This trained model then will be used to annotate unlabeled data. This soft labeled data would then be used to train a classifier. Am I getting it right?

For the train_examples, i believe input would look like this [CLS,....., [mask], SEP]. If this already have length of 512, with multiple token labels, this would output an error. Is there any way to limit the input size, so when i have multiple token labels, it would not exceed max length?

First of all, thanks for the huge effort.

I read your paper on Generative PET on ArXiv, you have mentioned that the source code is available on this repo, but it seems like you have not pushed it yet. I wonder if you are going to release the source code for Generative PET anytime soon.

Hi
I am not sure how to download the data for each task, could you kindly upload the files somewhere or include the links from which repo you got the data for each tasks?

Could you also comment how I can get unlabeled.jsonl for each task like boolq or rte?

thanks for your help

Hi,

It seems the auxiliary LM training includes the patterns. i.e. words in patterns also have the possibility of being masked. I wonder if there is any reason that you include the patterns in LM training, instead of using the original dataset. Many thanks.