Datasets in the benchmark with download links:

• CIFAR-100 (Image classification)
• Spherical CIFAR-100 (Transformed image classification) (272 MB)
• Ninapro DB5 (Hand-gesture classification)(15 MB)
• Darcy Flow (Partial differential equation solver) (1.6 GB)
• PSICOV (Protein sequence distance prediction) (1.1 GB)
• FSD50k (Sound event classification) (24 GB)
• Cosmic (Cosmic ray identification and replacement) (6.5 GB)
• ECG (Cardiac anomaly detection)(150 MB)
• Satellite (Earth monitoring through satellite imagery) (322 MB)
• DeepSEA (identifying chromatin features from DNA sequences)(860 MB)

Precomputed evaluation benchmark files on the NB201 search space (following NATS-Bench):

• NinaPro DB5(84 MB)
• Darcy Flow (85 MB)

Full outputs (include training logs):

• NinaPro DB5(46 GB)
• Darcy Flow (35.4 GB)

We use the open-source Determined software to implement experiment code.

Installing determined: pip install determined

A master instance is required:

• for local deployment (need to install docker):

  • to start the master: det deploy local cluster-up
  • access the WebUI at http://localhost:8080
  • to shut down: det deploy local cluster-down

• for AWS deployment (preferred):

  • install AWS CLI
  • Run aws configure and find AWS EC2 keypair name
  • to start the master: det deploy aws up --cluster-id CLUSTER_ID --keypair KEYPAIR_NAME
  • access the WebUI at {ec2-instance-uri}:8080
  • to shut down: det deploy aws down --cluster-id CLUSTER_ID

For an end-to-end example of running experiments with determined, you can refer to this video.

When running experiments, a docker image is automatically pulled from docker hub which contains all required python packages , i.e. you don't need to install them yourself, and it ensures reproducibility.

We provide pytorch implementations for two state-of-the-art NAS algorithms: GAEA PC-DARTS (paper link) and DenseNAS (paper link), which can be found inside each folder with the associated name, i.e. "darts/" for GAEA PC-DARTS and "densenas/" for DenseNAS.

To run these algorithms on 1D tasks, we've adapted their search spaces whose experiments are provided in "darts_1d/" for GAEA PC-DARTS (1D) and "densenas_1d/" for DenseNAS(1D).

Two task-specific NAS methods are implemented: Auto-DeepLab for dense prediction tasks in "autodeeplab/" and AMBER for 1D prediction tasks in "AMBER/".

We also implement procedure for running and tuning hyperparameters of the backbone architecture Wide ResNet (paper link), in "backbone/". The 1D-customized Wide ResNet is in "backbone_1d/".

To modify the random seed for each experiment, modify the number under

reproducibility: experiment_seed: for each script