I do not understand the log min max quantize quite well.
can you explain which bits are interger part and which are mantissa part after log min max quantize?
How can i get the interger representation of the quantized number?
Thank you so much.

does it work for conv2d_transpose?
I notice in quant.py, it works for nn.Conv2d, nn.Linear, nn.BatchNorm1d, nn.BatchNorm2d, nn.AvgPool2d. If I add nn.Conv2d_transpose, will it work?

I download the alexnet model from (https://download.pytorch.org/models/alexnet-owt-4df8aa71.pth) and evaluate the model by the val224_compressed.pkl with quantize.py , but the accuracy only 0.5397. Why?

when I run " python mnist/train.py",there is an error.

from utee import misc ModuleNotFoundError: No module named 'utee'

How can I fix it?

Hi, thanks for sharing the code. I run the training scripts of CIFAR100 but only get an accuracy of 66% (without quantization), instead of 74%. Can you please provide the parameter setting of the training script?

Thanks,

Hi,

when using log_minmax_quantize(), I found the bit depth is actually not 'bits' For example, based on the code, when the bits is 8, the actual quantization is using bits of 9. The function log_minmax_quantize() calls min_max_quantize() by passing 'bits'. With the extra 1 bit for 'sign', the total number of bit depth for quantization becomes bits+1.

feel free to correct me.

thanks,
jun

When I run this rounded = torch.floor(input / delta + 0.5), There is RuntimeError:
_th_floor_out is not implemented for type torch.cuda.LongTensor

I have been trying to access the quantized weights, but the quant layer has no attribute 'weight'. Looking at the code, I think that the quant layer only quantizes the input (i.e. the result of previous layers) in the forward pass, and not the weights of layers which I want quantized. Is there any workaround for this?

Is the code implemented following which paper or not?
Could u provide me with some related academic sources which guide the code writing?

    self.features = nn.Sequential(
        nn.Conv2d(3, 64, kernel_size=11, stride=4, padding=2),
        nn.ReLU(inplace=True),
        nn.MaxPool2d(kernel_size=3, stride=2),
        nn.Conv2d(64, 192, kernel_size=5, padding=2),
        nn.ReLU(inplace=True),
        nn.MaxPool2d(kernel_size=3, stride=2),
        nn.Conv2d(192, 384, kernel_size=3, padding=1),
        nn.ReLU(inplace=True),
        nn.Conv2d(384, 256, kernel_size=3, padding=1),
        nn.ReLU(inplace=True),
        nn.Conv2d(256, 256, kernel_size=3, padding=1),
        nn.ReLU(inplace=True),
        nn.MaxPool2d(kernel_size=3, stride=2),
    )

first code nn.Conv2d(3, 64, kernel_size=11, stride=4, padding=2), is right？
is not nn.Conv2d(3, 96, kernel_size=11, stride=4, padding=1)？

Pretrained models are not found at the links.

Hi, I have a modified vgg16 model. How do I specify that in the type flag for quantize.py given that I have my own path where my model is located.

when I run python quantize.py --type cifar10 --quant_method linear --param_bits 8 --fwd_bits 8 --bn_bits 8 --gpu 0. It prints Traceback (most recent call last):
File "quantize.py", line 64, in
sf = bits - 1. - quant.compute_integral_part(v, overflow_rate=args.overflow_rate)
File "/home/hjs/pytorch-playground-master/utee/quant.py", line 14, in compute_integral_part
v = v.data.cpu().numpy()[0]
IndexError: too many indices for array
I am looking for your help

Is this an issue on my end, or some issue with the download?

When I run python quantize.py --type cifar100 --model_root D:/graudate_design/pytorch-playground-master/models --data_root D:/graudate_design/pytorch-playground-master/cifar --gpu 0 --ngpu 1, I got two errors for idx, (data, target) in enumerate(tqdm.tqdm(ds, total=n_sample)): and misc.eval_model(model_raw, val_ds_tmp, ngpu=1, n_sample=args.n_sample, is_imagenet=is_imagenet)
in line 180 in \utee\misc.pyand in line 84 in \quantize.py.

Traceback (most recent call last):
File "/home/p/Work/pytorch-playground-master/quantize.py", line 102, in
main()
File "/home/p/Work/pytorch-playground-master/quantize.py", line 31, in main
args.gpu = misc.auto_select_gpu(utility_bound=0, num_gpu=args.ngpu, selected_gpus=args.gpu)
File "/home/p/Work/pytorch-playground-master/utee/misc.py", line 88, in auto_select_gpu
selected_gpus = selected_gpus.split(',')
AttributeError: 'int' object has no attribute 'split'

parser.add_argument('--gpu', default=1, help='index of gpus to use')
if default=1,the error will be appear
if default=1,the another error as fllows:

type: vgg16
quant_method: linear
batch_size: 24
gpu: ['1']
ngpu: 1
seed: 117
model_root: /home/p/.torch/models
data_root: /home/p/Work/pytorch-playground-master/data/train
logdir: log/default
input_size: 1024
n_sample: 20
param_bits: 8
bn_bits: 32
fwd_bits: 8
overflow_rate: 0.0

Setting GPU: ['1']
Traceback (most recent call last):
File "/home/p/Work/pytorch-playground-master/quantize.py", line 102, in
main()
File "/home/p/Work/pytorch-playground-master/quantize.py", line 43, in main
assert torch.cuda.is_available(), 'no cuda'
AssertionError: no cuda

what should i do?

import torch
from torch.autograd import Variable
from utee import selector
model_raw, ds_fetcher, is_imagenet = selector.select('svhn')
ds_val = ds_fetcher(batch_size=1, train=False, val=True)
for idx, (data, target) in enumerate(ds_val):
    data =  Variable(torch.FloatTensor(data)).cuda()
    output = model_raw(data)
    print(output.data.cpu().numpy())
    print(np.argmax(output.data.cpu().numpy()))
    print(target)

@aaron-xichen
I've tried this and the result was unsatisfactory ( it works fine for batch_size>1)
The followings are some results.

[[ 0.9298288 0.17785856 0.9966059 -1.3422316 -0.8245616 -1.9604906
-2.7374938 -1.6454206 -1.8076795 -2.0542176 ]]
2

4
[torch.LongTensor of size 1]

[[ 2.7919588 0.49071673 -0.00736207 -2.0971794 -0.84618115 -2.2232437
-1.6899112 -0.6246177 -2.9164875 -3.1203978 ]]
0

1
[torch.LongTensor of size 1]

[[ 2.9281554 -0.13904905 -1.3504897 -1.6061695 -2.2028518 -2.205429
-2.3030198 -1.7889079 -1.5594031 0.7322607 ]]
0

0
[torch.LongTensor of size 1]

[[ 1.6236477 -0.04049116 0.71252775 -1.5312613 -0.80628943 -1.1643314
-2.7663736 -1.779502 -1.764115 -2.6679206 ]]
0

9
[torch.LongTensor of size 1]

[[ 3.0169618 0.21566215 -0.64600635 -1.0604596 -2.5471652 -1.1154206
-3.167088 -1.7345388 -1.3325107 -1.2756087 ]]
0

5
[torch.LongTensor of size 1]

[[ 3.2620628 -1.1842312 -0.96141076 -0.72385985 -1.6753035 -0.01387847
-2.4336383 -2.6121674 -1.9076045 -2.078348 ]]
0

0
[torch.LongTensor of size 1]

[[ 3.155985 -0.25471076 -0.89448035 -0.77628857 -1.8204114 -3.392403
-2.5437825 -1.4555902 -1.0910527 -1.2623619 ]]
0

8
[torch.LongTensor of size 1]

[[ 2.4467583 -0.54981095 0.5102633 -1.2237269 -1.3451892 -2.4299471
-1.9993426 -1.2083414 0.04519886 -3.9493613 ]]
0

0
[torch.LongTensor of size 1]

[[ 3.0312405 0.00999061 -0.29210752 -1.6024964 -0.9401326 -1.0996346
-3.5482233 -0.65459347 -2.6797504 -2.7330062 ]]
0

0
[torch.LongTensor of size 1]

[[ 2.0508249 0.30808404 -0.6483562 -0.70096767 -1.3534812 -2.8651845
-2.9115884 -1.5140661 -0.58172 -1.2605368 ]]
0

7
[torch.LongTensor of size 1]

[[ 3.8504605 -1.934954 -0.1308189 -0.9230577 -1.7006387 -0.39881432
-3.0715973 -2.162218 -2.7872374 -1.0572728 ]]
0

2
[torch.LongTensor of size 1]

[[ 1.6195637 -2.3086352 -0.5728828 -0.86222374 0.429138 -0.6708926
-2.159588 -0.8012347 -2.1825366 -2.1155794 ]]
0

5
[torch.LongTensor of size 1]

[[ 1.5269842 -0.49805775 -0.44842458 -0.5922584 -2.7392595 -2.0895157
-1.2918607 -1.4767127 -1.2701623 -1.2878953 ]]
0

4
[torch.LongTensor of size 1]

[[ 1.8214152 -0.39846689 1.6661385 -0.7838743 -1.4924386 -1.0132811
-3.5224247 -1.7450149 -2.133103 -2.6991496 ]]
0

0
[torch.LongTensor of size 1]

[[ 3.3604963 -0.46506485 1.5834907 -3.2729144 -2.377539 -0.6647626
-3.1641421 -1.3123829 -1.9374268 -1.6866504 ]]
0

3
[torch.LongTensor of size 1]

[[ 1.7083087 1.0932902 -0.30605602 -0.89276785 -1.7285749 -1.6361132
-3.7625558 -1.6642044 -0.58260405 -2.6053936 ]]
0

3
[torch.LongTensor of size 1]

[[ 1.1230315 1.3429782 -0.7060108 -2.4044423 -1.0088397 -1.5541636
-0.94885933 -1.4849153 -1.6033367 -2.5828755 ]]
1

0
[torch.LongTensor of size 1]

[[ 1.9963155 -0.16534638 0.46809655 -1.1736008 -2.8804379 -0.66240954
-2.0466976 -2.3514454 -0.99034363 -1.7793173 ]]
0

5
[torch.LongTensor of size 1]

[[ 2.2264035 0.34485406 0.73350465 -1.9414191 -1.8649687 -3.4865122
-1.9899611 -1.1546313 -0.43400905 -2.308626 ]]
0

2
[torch.LongTensor of size 1]

I run this script,and it occur the error "print = misc.logger.info ^SyntaxError: invalid syntax",I don't know why.

It's really help me on the experiment.

For example my cnn based network named as “mycnn”.
How can I use “python quantize.py” to quantize “mycnn”？ Thank you and looking forward for your reply.

VGG(
(features): Sequential(
(0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(0_linear_quant): LinearQuant(sf=2, bits=8, overflow_rate=0.000, counter=0)
(1): ReLU(inplace=True)
(2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(2_linear_quant): LinearQuant(sf=1, bits=8, overflow_rate=0.000, counter=0)
(3): ReLU(inplace=True)
(4): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
(5): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(5_linear_quant): LinearQuant(sf=0, bits=8, overflow_rate=0.000, counter=0)
(6): ReLU(inplace=True)
(7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(7_linear_quant): LinearQuant(sf=0, bits=8, overflow_rate=0.000, counter=0)
(8): ReLU(inplace=True)
(9): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
(10): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(10_linear_quant): LinearQuant(sf=-1, bits=8, overflow_rate=0.000, counter=0)
(11): ReLU(inplace=True)
(12): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(12_linear_quant): LinearQuant(sf=-1, bits=8, overflow_rate=0.000, counter=0)
(13): ReLU(inplace=True)
(14): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(14_linear_quant): LinearQuant(sf=-1, bits=8, overflow_rate=0.000, counter=0)
(15): ReLU(inplace=True)
(16): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
(17): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(17_linear_quant): LinearQuant(sf=-1, bits=8, overflow_rate=0.000, counter=0)
(18): ReLU(inplace=True)
(19): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(19_linear_quant): LinearQuant(sf=-1, bits=8, overflow_rate=0.000, counter=0)
(20): ReLU(inplace=True)
(21): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(21_linear_quant): LinearQuant(sf=0, bits=8, overflow_rate=0.000, counter=0)
(22): ReLU(inplace=True)
(23): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
(24): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(24_linear_quant): LinearQuant(sf=0, bits=8, overflow_rate=0.000, counter=0)
(25): ReLU(inplace=True)
(26): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(26_linear_quant): LinearQuant(sf=0, bits=8, overflow_rate=0.000, counter=0)
(27): ReLU(inplace=True)
(28): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(28_linear_quant): LinearQuant(sf=0, bits=8, overflow_rate=0.000, counter=0)
(29): ReLU(inplace=True)
(30): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
)
(classifier): Sequential(
(0): Linear(in_features=25088, out_features=4096, bias=True)
(0_linear_quant): LinearQuant(sf=1, bits=8, overflow_rate=0.000, counter=0)
(1): ReLU(inplace=True)
(2): Dropout(p=0.5, inplace=False)
(3): Linear(in_features=4096, out_features=4096, bias=True)
(3_linear_quant): LinearQuant(sf=2, bits=8, overflow_rate=0.000, counter=0)
(4): ReLU(inplace=True)
(5): Dropout(p=0.5, inplace=False)
(6): Linear(in_features=4096, out_features=1000, bias=True)
(6_linear_quant): LinearQuant(sf=1, bits=8, overflow_rate=0.000, counter=0)
)
)

in features.0 layer, weights are 8 bits and ifmap is 32 bits.
Linear_quant layer is only generate after convolution layer so I don't know how the activation quantization works.
When do you quantize first input of NET?

when i run the command:
python3 quantize.py --type inception_v3 --quant_method linear --param_bits 8 --fwd_bits 8 --bn_bits 10 --ngpu 4 --batch_size 64 --seed 1
i got the error as below:
RuntimeError: The size of tensor a (3) must match the size of tensor b (864) at non-singleton dimension 3
library version:
torch-1.3.1+cu92 torchvision-0.4.2+cu92

could you help me?
thanks you!

when I print all of my ofmap of linear_quantization, the negative values are all zero even if they do not go through relu.
So, ReLU input = ReLU output right now.
Do you know why?

Thank you for you share, but when i run like below, i get result :acc1=0.00 acc5=0.00
''python quantize.py --type cifar10 --quant_method linear --param_bits 16 --fwd_bits 16 --bn_bits 16 --gpu 0 --ngpu 1 --input_size 32''

The first time I run, I get the following error,
''...not supported on CUDAType for Long''

then I change the data type to float, and then I get the above result. Thank you or your help.

There is something wrong, When I run the line for idx, (data, target) in enumerate(tqdm.tqdm(ds, total=n_sample)): in misc.py

Is there any reason why quantization is only applied if the module is a Sequential module and not everywhere? As in, the only place where I can see the activation quantization to not have an effect is if we do quantization before and after (though I don't know if floating point stability will affect this).

If I choose resnet50 or alexnet, it may be trained by imagenet.
Then what happend if I choose CIFAR-10 as modeltype? What net did I use for this option?

I think that there may be a problem with the way the package handles skip layers such as those found in ResNets.
My understanding is that the residual mapping F(x) + x seems to be quantized into:
Quantize(F(Quantize(x, s1)), s2) + Quantize(x, s1)
and ends up adding tensors that reside on two different scales. If I'm right, then the correctness of the output should still be fine (because the activations are re-scaled back to 2^sf and the computations carried out in float), but it would mean that the residual computation is done in more than the desired precision.

I am wondering how it performs the arithmetic operation for a quantized value.
That's say, 10-bit for input, 6-bit for weight, how does it perform a 10-bit x 6-bit multiplication?
Or maybe it doesn't do the "real" arithmetic operation?
(Simulate the value being quantized to 10-bit or 6-bit and dequantize them back to 16-bit float or 8-bit int, and use the 16-bit float or 8-bit int operation as usual?)

Thanks for your help!!

First I ran into an error in imagenet/inception.py, when initalizing the weights in the Inception3 network, the values created by truncated norm is just a 1-dim vector, it couldn't adjust itself to [in_channel, out_channel, kernel_size, kernel_size].
I solved it by view it as the shape of the kernel.

The second error is that using the BasicConv2d class to make conv operation with different parameters by passing in a dictionary. For example, the passed in parameter is {'kernel_size': 3, 'stride': 2}, but it got error:

RuntimeError: expected stride to be a single integer value or a list of 1 values to match the convolution dimensions, but got stride=[2, 2]
Shouldn't it be a list of stride going on with different dimension?

Thanks a lot for this nice utility to test quantization on pretrained models!

I'm trying to quantize the pretrained inception_v3 (directly from torchvision) by running the quantize.py script after minor modifications (to support external models).

From the code I see that the quant.duplicate_model_with_quant() function expects the model to contain a nn.Sequential block:

if isinstance(model, nn.Sequential):
    <add quantizer blocks after usual ops>

Since the torchvision inception_v3 model doesn't contain nn.Sequential, no _quant layers are not added in the final quantized model.

I wanted to understand what would be a clean fix retaining generality without losing the functionality you intended to keep by checking for nn.Sequential?

As of now, I'm just bypassing the if statement, but I presume that would add _quant blocks blindly to everything.

Output as it is:

DataParallel(
  (module): Inception3(
    (Conv2d_1a_3x3): BasicConv2d(
      (conv): Conv2d (3, 32, kernel_size=(3, 3), stride=(2, 2), bias=False)
      (bn): BatchNorm2d(32, eps=0.001, momentum=0.1, affine=True)
    )
...

Output with if statement bypassed:

Sequential(
  (module): Sequential(
    (Conv2d_1a_3x3): Sequential(
      (conv): Conv2d (3, 32, kernel_size=(3, 3), stride=(2, 2), bias=False)
      (conv_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=50)
      (bn): BatchNorm2d(32, eps=0.001, momentum=0.1, affine=True)
      (bn_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=50)
    )
...

It always predicts 464 for every sample...

import torch
import pickle as pkl
import time
import numpy as np
import cv2 
import matplotlib.pyplot as plt
import torchvision.models as models
import torchvision.transforms as transforms

def str2img(str_b):
    return cv2.imdecode(np.fromstring(str_b, np.uint8), cv2.IMREAD_COLOR)


def load_pickle(path):
    begin_st = time.time()
    with open(path, 'rb') as f:
        print("Loading pickle object from {}".format(path))
        v = pkl.load(f)
    print("=> Done ({:.4f} s)".format(time.time() - begin_st))
    return v

d = load_pickle('val224_compressed.pkl')

img224 = 0
target224 = 0
for img, target in zip(d['data'], d['target']):
    img224 = str2img(img)
    target224 = target
    break
    
normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])

img_tensor = transforms.ToTensor()(img224) / 255.
normalized_image = normalize(img_tensor)

model = models.resnet18(pretrained=True).eval()

pred = model(normalized_image.unsqueeze(0))

print(pred.argmax(1), target224)

Great work!

I'm trying to do some retraining to further aggravate quantization.
Are you willing to add any retraining modes in your scripts? Thanks!

From utee/quant.py, I can only find the process of adding quantized layer between different layer. But when it comes to quantized weight of conv, I can not find it.
So I wonder why you do not quant weight of conv, because it is importance in process of quantification.

python quantize.py --type alexnet --quant_method minmax --param_bits 8 --fwd_bits 8 --bn_bits 8 --ngpu 1

type=alexnet, quant_method=minmax, param_bits=8, bn_bits=8, fwd_bits=8, overflow_rate=0.0, acc1=0.5514, acc5=0.7819

print('save model...')
torch.save(model_raw.state_dict(), 'model_quantized.pth')
I am puzzled, after quantized, the model's memory are still 234M, how to save the quantied model？
Thanks for your help.

I find the parameters num_gpu=args.ngpu, selected_gpus=args.gpu by default they can be 0 and none,but when I run the train code, it would say Command 'nvidia-smi' returned non-zero exit status 127.. I reviewed the code, in the auto_select_gpu function, though no gpu selected, it would still try info = subprocess.check_output('nvidia-smi', shell=True).decode('utf-8'). so I wonder how can I continue in case of no GPU.

I executed quantization with MNIST dataset. After that, it returns acc1 and acc5. What's the difference between them? Why do you use them?

model_urls = {
'cifar10': 'http://ml.cs.tsinghua.edu.cn/~chenxi/pytorch-models/cifar10-d875770b.pth',
'cifar100': 'http://ml.cs.tsinghua.edu.cn/~chenxi/pytorch-models/cifar100-3a55a987.pth',
}

model url 404 not found

This is a playground for pytorch beginners, which contains predefined models on popular dataset. Currently we support

mnist, svhn
cifar10, cifar100
stl10
alexnet
vgg16, vgg16_bn, vgg19, vgg19_bn
resent18->resnet18, resent34->resnet34, resnet50, resnet101, resnet152
squeezenet_v0, squeezenet_v1
inception_v3

I tried quantizing a VGG-16 network but the size of the network hasn't changed. I loaded a VGG-16 model from torchvision and ran a portion of the quantize.py (lines 48 - 79) which does the quantization. When I saved the model - the size had increased slightly.

Can you please tell me what I am doing incorrectly?

I run convert.py. After loading to 100%, it will take a long time.

I got some trouble in load pre-train models.
I can't download the resnet50 pre-train models by the link,such as:https://download.pytorch.org/models/resnet50-19c8e357.pth

By the way ,I'm in china,is this the reason for this?

Pretty work, can be further provided Top-5 Accuracy?

FYI.
TF-slim:
https://github.com/tensorflow/models/tree/master/slim

Thank you for providing the models.
Could you please check the pretrained SVHN?
The accuracy of the pretrained SVHN is close to zero.

Hi,
When I was trying to using the Alexnet model, I first of all tried to follow your instruction to download val224_compressed.pkl and executed the command "python convert.py"
But when I was converting, it always come to the error message "Memory Error".
I am curious about how to deal with this issue, since I think the memory of the machine I used is big enough, which is 64 GB.
Thanks !

RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

could you tell me the pytorch version?