This repository contains public releases of SenseTime Helios traces for the benefit of the deep learning system research community.

If you do use the Helios traces in your research, please make sure to cite our SC '21 paper "Characterization and Prediction of Deep Learning Workloads in Large-Scale GPU Datacenters", which includes a comprehensive analysis of the deep learning workloads in Helios from April 2020 to September 2020.

We encourage anyone to use the traces for academic purposes, and if you had any questions, feel free to send an email to us, or file an issue on Github.

Helios is a private datacenter dedicated to developing DL models for research and production in SenseTime. It contains 8 independent GPU clusters and over 12,000 GPUs in total.

In this repository, we publicly release the workload trace in 4 representative GPU clusters: Earth, Saturn, Uranus, and Venus. You can find a detailed description of the Helios datacenter in the SC '21 paper mentioned above.

Besides, we also release the analysis scripts for Helios traces in HeliosArtifact.

The main trace characteristics, dataset structure and schema are:

• Dataset size: 343MB
• Compressed dataset size: 36.4MB
• Number of files: 8
• Duration: 6 months
• Number of independent GPU clusters: 4
• Total number of jobs: 3,362,981
• Total number of GPU jobs: 1,580,464

Each cluster provides a job trace file (cluster_log.csv) and a VC configuration file (cluster_gpu_number.csv).

📦data
 ┣ 📂Earth
 ┃ ┣ 📜cluster_gpu_number.csv
 ┃ ┗ 📜cluster_log.csv
 ┣ 📂Saturn
 ┃ ┣ 📜cluster_gpu_number.csv
 ┃ ┗ 📜cluster_log.csv
 ┣ 📂Uranus
 ┃ ┣ 📜cluster_gpu_number.csv
 ┃ ┗ 📜cluster_log.csv
 ┗ 📂Venus
 ┃ ┣ 📜cluster_gpu_number.csv
 ┃ ┗ 📜cluster_log.csv

Provides rich information on all jobs submitted to Slurm in each cluster.

1. job_id is generated by Slurm and reflects the job submission order in each cluster.
2. A job can end up with one of five statuses: (1) COMPLETED: it is finished successfully; (2) CANCELLED: it is terminated by the user; (3) FAILED: it is terminated due to internal or external errors; (4) TIMEOUT: the execution time is out of limit; (5) NODE_FAIL: it is terminated due to the node crash. TIMEOUT and NODE_FAIL are very rare in our traces, and are regarded as failed in our analysis. (Another status SUSPENDED happens only once in cluster Uranus, so we ignore it.)
3. Calculated from the difference between end_time and start_time. (Unit: seconds)
4. Calculated from the difference between start_time and submit_time. (Unit: seconds)

Lists the number of GPUs per day in each VC.