veiled-toxicity-detection's Introduction

Veiled Toxicity Detection

This repo contains code for our paper Fortifying Toxic Speech Detectors Against Veiled Toxicity (https://arxiv.org/abs/2010.03154).

Setup

The main environment requirements for this project are python 3.6, pytorch 1.3.1, and pytorch-pretrained-bert 0.6.1.

Data processing

We mainly use three sources of data (as describe in our paper): the Social Bias Inference Corpus [1], a random crawl of Reddit, and queries to the Perspective API [2].

resources/process_SBF.ipynb: filters the SBIC dataset by their offensiveness and Perspective API toxicity scores (described in the paper).
resources/processed_dataset/*.pkl: pickle files containing the processed datasets (format: List of (original_text, (target_group_annotations)))

Training the initial student model

model/*-aws-run-bert.sh: trains a base classifier; this classifier approximates the behavior of Perspective API on our processed datasets.

Applying different influence metrics

bert_embin.py: implements the embedding similarity metric.
bert_influence.py: implements the influence functions metric.
bert_trackin.py: implements the gradient product (TrackIn) metric.
*-aws-run-inf.sh: calculates the influence of each training example on each probing example using different influence metrics.

Training the fortified model

analysis.ipynb: visualizes influence as presented in the paper, and marks data to relabel.
model/*-aws-correct-bert.sh: trains an improved classifier with re-labeled influential training examples.

If you have any questions, please email Xiaochuang Han at [email protected]. Thank you!

[1] Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A Smith & Yejin Choi. Social Bias Frames: Reasoning about Social and Power Implications of Language. ACL (2020)

[2] https://www.perspectiveapi.com/#/home

veiled-toxicity-detection's People