DMS(Data-Mutation based Selection) is a test case selection method. We conduct experiments on five pairs of widely-used deep learning test sets and models. The results show that DMS significantly outperforms existing test case selection methods in terms of both bug-revealing ability and diversity of bug-revealing direction. Specifically, taking the original test set as the candidate set, DMS can filter out 53.85% to 99.22% of all bug-revealing test cases when selecting 10% of the test cases. Moreover, when selecting 5% of the test cases, the selected cases can cover almost all bug-revealing directions across all subjects. Overall, DMS outperforms baseline approaches with an average improvement of 12.38% to 71.81% in terms of the bug-revealing test cases selected, which further demonstrates the effectiveness of DMS.

• numpy 1.16.4
• tensorflow 1.14.0
• tqdm 4.36.1
• h5py 2.10.0
• pandas 0.23.4

Please use the following command to generate mutated models.

python finetune.py -d mnist -n lenet5 -i 40 -e 25

Please use the folling command to obtain prediction results of mutated models.

python finetune_predict.py -d mnist -n lenet5

Please use the folling command to extract features of test cases.

python extract_info.py -d mnist -n lenet5

Please use the folling command to obtain results of all test case selection methods.

python demo_general.py -d mnist -n lenet5