coarse grain the network

• retry with tanh and relu and see if i get zero train loss

do shuffling svd exps & pca regression:

MNIST: Classification with nonlinearity
Try experiment with L2 set to what we have now and an experiment without L2
Start with easy nonlinearities like tanh and then try ReLU

for k in {1..14}; do python -u mnist_classification.py ./data --coarsegrain_blocksize $k --num_hidden_features $i --num_train_samples $j --fileprefix tanhlr0.1_mnistAUG05 --nonlinearity tanh --train_method gradient_descent --epochs 150 --lr 0.1 ; done

for k in {1..14}; do python -u mnist_classification.py ./data --coarsegrain_blocksize $k --num_hidden_features $i --num_train_samples $j --fileprefix tanhlr0.1_wd0_mnistAUG06 --nonlinearity tanh --train_method gradient_descent --epochs 150 --lr 0.1 --wd 0.0000000000000000000000001; done

for k in 1 2 3 4 5 6 7 8 9 11 17; do python -u cifar_classification_randomfeatures.py ./data  --coarsegrain_blocksize  $k --num_hidden_features $i --num_train_samples $j  --fileprefix linelr0.001_wd1em5_cifarAUG15 --nonlinearity line --train_method gradient_descent --epochs 150 --lr 0.001; done #    --wd 0.0000000000000000000000001

for k in 1 2 3 4 5 6 7 8 9 11 17; do python -u cifar_classification_randomfeatures.py ./data  --coarsegrain_blocksize  $k --num_hidden_features $i --num_train_samples $j  --fileprefix tanhlr0.1_wd1em5_cifarAUG15 --nonlinearity tanh --train_method gradient_descent --epochs 150 --lr 0.1; done #    --wd 0.0000000000000000000000001

Whole network

for k in 1 2 3 4 5 6 7 8 9 11 17; do python -u cifar_classification_randomfeatures_wholenetwork.py ./data  --coarsegrain_blocksize  $k --num_hidden_features $i --num_train_samples $j  --fileprefix linelr0.1_wd1em5_cifar_fullnetAUG16 --nonlinearity line--train_method gradient_descent --epochs 150 --lr 0.1; done #    --wd 0.0000000000000000000000001


for k in 1 2 3 4 5 6 7 8 9 11 17; do python -u cifar_classification_randomfeatures_wholenetwork.py ./data  --coarsegrain_blocksize  $k --num_hidden_features $i --num_train_samples $j  --fileprefix tanhlr0.1_wd1em5_cifar_fullnetAUG16 --nonlinearity tanh --train_method gradient_descent --epochs 150 --lr 0.1; done #    --wd 0.0000000000000000000000001

for k in 1 2 3 4 5 6 7 8 9 11 17; do python -u cifar_classification_randomfeatures_wholenetwork.py ./data  --coarsegrain_blocksize  $k --num_hidden_features $i --num_train_samples $j  --fileprefix relulr0.001_wd1em5_cifar_fullnetAUG16 --nonlinearity relu --train_method gradient_descent --epochs 150 --lr 0.001; done #    --wd 0.0000000000000000000000001

principle component with decorrelated

shuffle the values of principal components:

each sample now has its

instead of zeroing them out, independently shuffle across all data points, keep the top k dimensions
make a plot of no. of dimensions kept

for k in 28 27 25 23 21 19 17 15 13 11 9 7 5 3 1; do python -u mnist_classification_wholenetwork.py ./data --target_size $k --num_hidden_features $j --num_train_samples 60000 --fileprefix relulr0.1_mnist_fullnet_fractional_cg_DEC04 --lr 0.1 --nonlinearity relu --epochs 150 --upsample; done # --wd 0.0000000000000000000000001

for k in 28 27 25 23 21 19 17 15 13 11 9 7 5 3 1; do python -u mnist_classification_wholenetwork.py ./data --target_size $k --num_hidden_features $j --num_train_samples 60000 --fileprefix linelr0.1_mnist_fullnet_fractional_cg_DEC04 --lr 0.1 --nonlinearity relu --epochs 150 --upsample; done # --wd 0.0000000000000000000000001