This project depends on https://pypi.org/project/emoji/ and https://pypi.org/project/clean-text/ for the preprocessing of the data.

Normalise the list of files; train and test:

python3 read_data.py 
                     --to_normalise LIST_OF_FILES
                     --normalised_path LIST_OF_NORMALISED_PATHS_IN_ORDER
                     --language en

Concatenate files:

python3 read_data.py 
                     --normalised_path LIST_OF_NORMALISED_PATHS
                     --concat CONCATENATED_PATH

For this project we used the following BERT models:

• https://deepset.ai/german-bert,
• https://huggingface.co/bert-base-uncased, and
• https://huggingface.co/bert-base-multilingual-cased

The optimal number of epochs for each model were the following:

• EN: 2,
• EN-multi: 2,
• DE: 1,
• DE-multi: 3,
• DE-HASOC: 2,
• DE-HASOC-multi: 8,
• GermEval: 6,
• GermEval-multi: 4

Change the data paths in the config files to direct to the concatenated normalised path and the test.json to direct to the normalised test files.

Important note: if the data_path parameter is a single file, it will be split into train and validation. If you want the model to train on the whole data, you have to make it the first element of a list, like you can see in the configs/English_train_whole_data.json file.

python3 run_configs.py 
                     --mode train
                     --configs ./configs 
                     --test_files test.json

After the training process finished, the best systems will give a prediction on the given test files. These will be put into the predicted dictionary. The training process is the same for the categorical subtask.

Run the following script to get the final result:

python3 run_configs.py 
                     --mode result
                     --configs ./configs 
                     --test_files test.json

If the test file does not contain labels, add the --test argument to the above command.

Find the rule systems under scripts/rule_system