We choose the FRGC dataset (See challenge) as it is more popular according to the number of citations, so more related work to look into.
For bonafide_*
:
- Example image name:
04927d04.png
- Unique identifier of length 5:
04927
- Session identifier(?):
d04
For morphs_*
:
e.g.
02463d252.png_vs_04494d50 02463d632.png_vs_04211d214
find . -type f -name '*02463*'
# 4 morphs
# ./02463d252.png_vs_04494d50.png
# ./02463d632.png_vs_04211d214.png
# ./04463d155.png_vs_02463d252.png
# ./04929d18.png_vs_02463d632.png
# *morphID* *subjectID* *score1* *score2* ... *scoreN*
M0001 S1 0.2838268129361562 0.3155098670283303 0.27628365453527004 0.28536534857627505 0.31393568652389425 0.27158329983997165 0.251354142718645 0.274077943997888 0.236568918377147 0.25457697711437777
Warning! Some images have less probe images than others.
- ArcFace+yunet (one of the already used in the MAP literature)
- Facenet512+retinaface (the one with best performance + state of the art)
The choices were made according to the table found here: Benchmarks on choice of distance metric, alignment, model and detector
To calculate the appropriate thresholds, pyeer was used. We wanted the thresholds to satisfy this condition: The False Match Rate must be equal to 0.1%.
This library was also helpful to calculate other scores and plot DET curves.
We calculate MMPMR, RMMR scores to compare with the MAP metric.
The instructions are provided for experiment reproducability.
The Input files are not provided for privacy reasons.
The root Input
folder structure should look like this:
Input
├── Database
│ ├── bonafide_probe
│ ├── bonafide_reference
│ ├── morphs_facefusion
│ ├── morphs_facemorpher
│ ├── morphs_opencv
│ └── morphs_ubo
First clone the repository
git clone --recurse-submodules https://github.com/Mirtia/Deepface-MAP-Metrics.git
Create a python environment using miniconda
or your python environment manager of your choice.
Install requirements using pip or any python package manager:
pip install -r requirements.txt
To analyze the databases and calculate the dissimilarity scores as well as the mated and non-mated scores:
python src/main.py -i Input/ -o Output -m deepface
Next, to calculate the metrics from the mated and non-mated files, install pyeer
:
pip install pyeer
The score files were cleaned:
cat FRGC_ArcFace+yunet_mated_scores.txt | awk '{print $2}' > AY_mated_scores.txt
cat FRGC_Facenet512+retinaface_mated_scores.txt | awk '{print $2}' > FR_mated_scores.txt
cat FRGC_ArcFace+yunet_non_mated_scores.txt | awk '{print $3}' > AY_non_mated_scores.txt
cat FRGC_Facenet512+retinaface_non_mated_scores.txt | awk '{print $3}' > FR_non_mated_scores.txt
# For other FRSs
cat FRGC_Facenet+yunet_mated_scores.txt | awk '{print $2}' > FY_mated_scores.txt
cat FRGC_VGG-Face+yunet_mated_scores.txt | awk '{print $2}' > VGG_mated_scores.txt
cat FRGC_Facenet+yunet_non_mated_scores.txt | awk '{print $3}' > FY_non_mated_scores.txt
cat FRGC_VGG-Face+yunet_non_mated_scores.txt | awk '{print $3}' > VGG_non_mated_scores.txt
cat FRGC_VGG-Face+mtcnn_mated_scores.txt | awk '{print $2}' > VGGm_mated_scores.txt
cat FRGC_VGG-Face+mtcnn_non_mated_scores.txt | awk '{print $3}' > VGGm_non_mated_scores.txt
Then, run pyeer! Do not forget -ds
for dissimilarity scores. Pyeer can run at the original files as well, as it only considers the last column for scores.
geteerinf -p . -g AY_mated_scores.txt -i AY_non_mated_scores.txt -e "AY-Output" -ds
geteerinf -p . -g FR_mated_scores.txt -i FR_non_mated_scores.txt -e "FR-Output" -ds
geteerinf -p . -g FY_mated_scores.txt -i FY_non_mated_scores.txt -e "FY-Output" -ds
geteerinf -p . -g VGG_mated_scores.txt -i VGG_non_mated_scores.txt -e "VGG-Output" -ds
pyeer
computes for you the threshold values (see FMR1000_TH
) which can be found at 1.pyeer_report.csv and at 2.pyeer_report.csv
The columns of the report files were formatted for better readability.
To calculate MMPMR, RMMR... scores, provide the files and threshold (can be found in FRS_info.json
:
# For FRS_1 = ArcFace+yunet
python src/main.py -g Metrics/AY_mated_scores.txt -n Metrics/AY_non_mated_scores.txt -t 0.542889 -m mr
# FOR FRS_2 = Facenet512+retinaface
python src/main.py -g Metrics/FR_mated_scores.txt -n Metrics/FR_non_mated_scores.txt -t 0.333671 -m mr
To plot the Negative and Positive Rates and generate the confusion matrices, the DET Scripts provided by the course instructors were used.