MLAgility offers a complementary approach to MLPerf by examining the capability of vendors to provide turnkey solutions to a corpus of hundreds of off-the-shelf models. All of the model scripts and benchmarking code are published as open source software. The performance data is available at our Huggingface Space.

Our benchit CLI allows you to benchmark Pytorch models without changing a single line of code. The demo below shows BERT-Base being benchmarked on both Nvidia A100 and Intel Xeon. For more information, check out our Tutorials and Tools User Guide.

You can reproduce this demo by trying out the Just Benchmark BERT tutorial.

This repository is home to a diverse corpus of hundreds of models. We are actively working on increasing the number of models on our model library. You can see the set of models in each category by clicking on the corresponding badge.

We are also working on making MLAgility results publicly available at our Huggingface Space. Check it out!

The diagram above illustrates the MLAgility repository's structure. Simply put, the MLAgility models are leveraged by our benchmarking tool, benchit, to produce benchmarking outcomes showcased on our Hugging Face Spaces page.

Please refer to our mlagility installation guide to get instructions on how to install mlagility.

We are actively seeking collaborators from across the industry. If you would like to contribute to this project, please check out our contribution guide.

This project is licensed under the MIT License - see the LICENSE file for details.