kranthigv / bias-evaluation Goto Github PK

• We will be dealing with several models (BERT, RoBERTa, DistilBERT, etc).
• So, it would be simpler if we can define a protocol that all the model implementations respect.
• This enables us to evaluate all the models with a unified interface.

Other thoughts:

• We may need different types of protocols for different evaluation benchmarks.

• StereoSet is available here: https://stereoset.mit.edu
• Code to reproduce their experiments is available here: https://github.com/moinnadeem/StereoSet
• We need to import this evaluation strategy in our codebase.

Reference: https://aclanthology.org/N19-1063.pdf

From the repo: https://github.com/nyu-mll/crows-pairs