• Java 1.8
• Maven
• R and Rscript

In your R console type install.packages("irr") to install 'irr' package.

To compile the script type the following commands:

git clone https://github.com/johnnymoretti/CoNLL-U_Fleiss_Kappa.git
cd CoNLL-U_Fleiss_Kappa
mvn package

After the setup procedure it's possible to use the script with:

java -jar target/CoNLL-U_Fleiss_irr.jar -m POS|MORPH|EDGES|DEPREL <conllu file rate 1> <conllu file rate 2> ...

The 'mode' parameter specify the ConLL-U field used in irr evaluation. It's possible to choose among POS,EDGES and DEPREL.