We choose the FRGC dataset (See challenge) as it is more popular according to the number of citations, so more related work to look into.
For bonafide_*:
- Example image name:
04927d04.png - Unique identifier of length 5:
04927 - Session identifier(?):
d04
For morphs_*:
e.g.
02463d252.png_vs_04494d50 02463d632.png_vs_04211d214
find . -type f -name '*02463*'
# 4 morphs
# ./02463d252.png_vs_04494d50.png
# ./02463d632.png_vs_04211d214.png
# ./04463d155.png_vs_02463d252.png
# ./04929d18.png_vs_02463d632.png# *morphID* *subjectID* *score1* *score2* ... *scoreN*
M0001 S1 0.2838268129361562 0.3155098670283303 0.27628365453527004 0.28536534857627505 0.31393568652389425 0.27158329983997165 0.251354142718645 0.274077943997888 0.236568918377147 0.25457697711437777Warning! Some images have less probe images than others.
- ArcFace+yunet (one of the already used in the MAP literature)
- Facenet512+retinaface (the one with best performance + state of the art)
The choices were made according to the table found here: Benchmarks on choice of distance metric, alignment, model and detector
To calculate the appropriate thresholds, pyeer was used. We wanted the thresholds to satisfy this condition: The False Match Rate must be equal to 0.1%.
This library was also helpful to calculate other scores and plot DET curves.
We calculate MMPMR, RMMR scores to compare with the MAP metric.
The instructions are provided for experiment reproducability.
The Input files are not provided for privacy reasons.
The root Input folder structure should look like this:
Input
├── Database
│ ├── bonafide_probe
│ ├── bonafide_reference
│ ├── morphs_facefusion
│ ├── morphs_facemorpher
│ ├── morphs_opencv
│ └── morphs_uboFirst clone the repository
git clone --recurse-submodules https://github.com/Mirtia/Deepface-MAP-Metrics.gitCreate a python environment using miniconda or your python environment manager of your choice.
Install requirements using pip or any python package manager:
pip install -r requirements.txtTo analyze the databases and calculate the dissimilarity scores as well as the mated and non-mated scores:
python src/main.py -i Input/ -o Output -m deepfaceNext, to calculate the metrics from the mated and non-mated files, install pyeer:
pip install pyeerThe score files were cleaned:
cat FRGC_ArcFace+yunet_mated_scores.txt | awk '{print $2}' > AY_mated_scores.txt
cat FRGC_Facenet512+retinaface_mated_scores.txt | awk '{print $2}' > FR_mated_scores.txt
cat FRGC_ArcFace+yunet_non_mated_scores.txt | awk '{print $3}' > AY_non_mated_scores.txt
cat FRGC_Facenet512+retinaface_non_mated_scores.txt | awk '{print $3}' > FR_non_mated_scores.txt
# For other FRSs
cat FRGC_Facenet+yunet_mated_scores.txt | awk '{print $2}' > FY_mated_scores.txt
cat FRGC_VGG-Face+yunet_mated_scores.txt | awk '{print $2}' > VGG_mated_scores.txt
cat FRGC_Facenet+yunet_non_mated_scores.txt | awk '{print $3}' > FY_non_mated_scores.txt
cat FRGC_VGG-Face+yunet_non_mated_scores.txt | awk '{print $3}' > VGG_non_mated_scores.txt
cat FRGC_VGG-Face+mtcnn_mated_scores.txt | awk '{print $2}' > VGGm_mated_scores.txt
cat FRGC_VGG-Face+mtcnn_non_mated_scores.txt | awk '{print $3}' > VGGm_non_mated_scores.txtThen, run pyeer! Do not forget -ds for dissimilarity scores. Pyeer can run at the original files as well, as it only considers the last column for scores.
geteerinf -p . -g AY_mated_scores.txt -i AY_non_mated_scores.txt -e "AY-Output" -ds
geteerinf -p . -g FR_mated_scores.txt -i FR_non_mated_scores.txt -e "FR-Output" -ds
geteerinf -p . -g FY_mated_scores.txt -i FY_non_mated_scores.txt -e "FY-Output" -ds
geteerinf -p . -g VGG_mated_scores.txt -i VGG_non_mated_scores.txt -e "VGG-Output" -dspyeer computes for you the threshold values (see FMR1000_TH) which can be found at 1.pyeer_report.csv and at 2.pyeer_report.csv
The columns of the report files were formatted for better readability.
To calculate MMPMR, RMMR... scores, provide the files and threshold (can be found in FRS_info.json:
# For FRS_1 = ArcFace+yunet
python src/main.py -g Metrics/AY_mated_scores.txt -n Metrics/AY_non_mated_scores.txt -t 0.542889 -m mr
# FOR FRS_2 = Facenet512+retinaface
python src/main.py -g Metrics/FR_mated_scores.txt -n Metrics/FR_non_mated_scores.txt -t 0.333671 -m mrTo plot the Negative and Positive Rates and generate the confusion matrices, the DET Scripts provided by the course instructors were used.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent