This repository contains the code to train neural nets and compute various measures/norms reported in the following paper:

Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks

Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, Nathan Srebro

1. Install Python 3.10 and PyTorch 2.2.2.

   conda create -n myenv python=3.8
   conda activate myenv
   conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia (if use cpu: conda install pytorch torchvision torchaudio cpuonly -c pytorch)
   

2. Clone the repository:

   git clone https://github.com/Gear-dev-sudo/CNN_test.git
   cd CNN_test
   

3. As a simple example, the following command trains a two layer fully connected feedforward network with 1000 hidden units on CIFAR10 dataset and then computes several measures/norms on the learned network:

   python main.py --dataset CIFAR10 --epochs 20
   

• --no-cuda: disables cuda training
• --datadir: path to the directory that contains the datasets (default: datasets)
• --dataset: name of the dataset(options: MNIST | CIFAR10 | CIFAR100 | SVHN, default: CIFAR10). If the dataset is not in the desired directory, it will be downloaded.
• --nunits: number of hidden units (default: 1024)

After training the network, several norms/measures will be computed and reported on the trained network. Please see the file measures.py for explanation of each measure. We also compute and report the following generalization bounds:

• VC bound: Generalization bound based on the VC dimension by Harvey et al. 2017
• L1max bound: Generalization bound by Bartlett and Mendelson 2002
• Fro bound: Generalization bound by Neyshabur et al. 2015
• Spec_L1 bound: Generalization bound by Bartlett et al. 2017
• Spec_Fro bound: Generalization bound by Neyshabur et al. 2018
• Our bound: The Generalization bound proposed in this paper