I use the tools to count FLOPs for EXTD-pytorch
The FLOPs result is different from the result computed by EXTD-pytorch which has code for computing flops as follows:

def compute_flops(model, image_size):
  import torch.nn as nn
  flops = 0.
  input_size = image_size
  for m in model.modules():
    if isinstance(m, nn.AvgPool2d) or isinstance(m, nn.MaxPool2d):
      input_size = input_size / 2.
    if isinstance(m, nn.Conv2d):
      if m.groups == 1:
        flop = (input_size[0] / m.stride[0] * input_size[1] / m.stride[1]) * m.kernel_size[0] ** 2 * m.in_channels * m.out_channels
      else:
        flop = (input_size[0] / m.stride[0] * input_size[1] / m.stride[1]) * m.kernel_size[0] ** 2 * ((m.in_channels/m.groups) * (m.out_channels/m.groups) * m.groups)
      flops += flop
      if m.stride[0] == 2: input_size = input_size / 2.

  return flops / 1000000000., flops / 1000000

The result by OpCounter is 1.084G, while 11.15G by above code.
Input size is 640x640

What's reason of this??

python 2.7.12
python 3.6.7
ubuntu 16.04 64bit

I have a model which is needed to be profiled with inputs in different sizes. I tested the following example code in jupyter notebook, where the second run without reloading the model threw a warning and no result returned:

import sys
pyversion = sys.version.split(' ')[0]
print('current python version: {:s}'.format(pyversion))

>> current python version: 3.7.2

import torch
torchversion = torch.__version__
print('current pytorch version: {:s}'.format(torch.__version__))

import torchvision
from torchvision import models
print('current torchvision version: {:s}'.format(torchvision.__version__))

>> current pytorch version: 1.1.0.post2
>> current torchvision version: 0.3.0

# VGG16
vgg16 = models.vgg16()
from thop import profile
inp = torch.randn(1, 3, 224, 224, )
profile(vgg16, inputs=(inp, ))

>> .....(normal output, too long to list)

inp = torch.randn(1, 3, 448, 448, )
profile(vgg16, inputs=(inp, ))

---------------------------------------------------------------------------
Warning                                   Traceback (most recent call last)
<ipython-input-4-9bd5a4920d3c> in <module>
      1 inp = torch.randn(1, 3, 448, 448, )
----> 2 profile(vgg16, inputs=(inp, ))

/usr/local/lib/python3.7/site-packages/thop/profile.py in profile(model, inputs, custom_ops, verbose)
     67 
     68     model.eval()
---> 69     model.apply(add_hooks)
     70 
     71     with torch.no_grad():

/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py in apply(self, fn)
    245         """
    246         for module in self.children():
--> 247             module.apply(fn)
    248         fn(self)
    249         return self

/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py in apply(self, fn)
    245         """
    246         for module in self.children():
--> 247             module.apply(fn)
    248         fn(self)
    249         return self

/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py in apply(self, fn)
    246         for module in self.children():
    247             module.apply(fn)
--> 248         fn(self)
    249         return self
    250 

/usr/local/lib/python3.7/site-packages/thop/profile.py in add_hooks(m)
     38         if hasattr(m, "total_ops") or hasattr(m, "total_params"):
     39             raise Warning("Either .total_ops or .total_params is already defined in %s.\n"
---> 40                           "Be careful, it might change your code's behavior." % str(m))
     41 
     42         m.register_buffer('total_ops', torch.zeros(1))

Warning: Either .total_ops or .total_params is already defined in Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)).
Be careful, it might change your code's behavior.

Since I have a lot of inputs in different sizes (like images in a dataset), it is much slower if I reload the model everytime an input comes in. Is this a PyTorch design feature or this repo's? Is there any workarounds?

Hi Lyken, thank you for your powerful tool, however, I met some difficult with it and get error as

'torch.Size' object has no attribute 'numel'

My environment is python 3.6 with torch 0.4.1.

First of all, thanks to the author's selfless dedication, your code is well written!
I have implemented two versions of 3D CNN(separable 3D and P3D) and want to calculate their FLOPs. I found that the FLOPs of separable 3D is much smaller than P3D version using your code (283.237G FLOPs VS 3.246T FLOPs). But when I calculate the inference time averaging in 100 forwards, the P3D version is much faster (0.8194s VS 0.3823s). Is there any problem when calculating computation of 3D CNN?

It seems that thop only support python3.
In python2.7, there will be syntax error.

thop-0.0.22-py2.7.egg/thop/count_hooks.py, line 119
kernel = torch.Tensor([*(x[0].shape[2:])]) // torch.Tensor(list((m.output_size,))).squeeze()
SyntaxError: invalid syntax

logs:
Either .total_ops or .total_params is already defined in Conv2d(3, 16, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False).Be careful, it might change your code's beh avior. Either .total_ops or .total_params is already defined in BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True).Be careful, it might change your code's behavior.

It has multiple errors:

• profile.py", line 70, in profile
  original_device = model.parameters().next().device
  AttributeError: 'generator' object has no attribute 'next'

If I get my own network from some baseline models which is modified somewhat, and how can I use the thop, can you give me an example for reference, thanks a lot

The original output of profile is too ugly and it seems that you add a visualization tool in utils

However, the clever_format tool is not included in the __init__ so it can not be smoothly called. My recommendation is add an clever_format in your profile arguments, and make it True by default.

Traceback (most recent call last):
File "/home/gpu/chen/IMIXNet-PyTorch/code/get_Netpara.py", line 21, in
print("Torch:", thop.profile(torch.nn.Conv2d(1, 128, (3, 3)), inputs=(torch.zeros((1, 1, 128, 128)),), verbose=False)[0])
File "/home/gpu/.conda/envs/pytorch4/lib/python3.6/site-packages/thop/profile.py", line 92, in profile
model(*inputs)
File "/home/gpu/.conda/envs/pytorch4/lib/python3.6/site-packages/torch/nn/modules/module.py", line 479, in call
hook_result = hook(self, input, result)
File "/home/gpu/.conda/envs/pytorch4/lib/python3.6/site-packages/thop/count_hooks.py", line 20, in count_convNd
kernel_ops = m.weight.size()[2:].numel() # Kw x Kh
AttributeError: 'torch.Size' object has no attribute 'numel'

what‘s the problem here ?
the version of thop is wrong?

In my model, nn.functional.interpolate is used, whose computation should be also calculated in my case.

nn.functional.interpolate: count_upsample_bilinear was appended in register_hooks in profile.py, and count_upsample_bilinear was implemented in count_hooks.py, but it did not work.

I also tried to print m_type in add_hooks in profile, but the output did not include nn.functional.interpolate. It seems that we cannot add hooks to some APIs like nn.functional.interpolate which work differently from APIs from torch.nn in terms of parameters and handler. I wonder if we can only wrap them into 3rd party modules to set FLOPs counters for them like the example in README.md.

Is there any workarounds introducing minimum modification to the model definition code?

Hello！
I want to use the code to count the Multi-Adds operations, but I don't know if the Multi-Adds operations and MACs and FLOPs are the same thing.
I'm confused about the definition of MACs and FLOPs, and I'm also not clearly know if the code you provided is to count the MACs or FLOPs .
Could you tell me the definition of them and the difference between them?
Thank you!

Hi. What environment is needed to install THOP? When I run the command 'pip install thop', it shows 'ERROR: No matching distribution found for thop'. Thanks.

Could you please give a closer explanation for the calculation of the AvgPool FLOPs? Despite I know that the impact of pooling layers is minimal, I'd like to understand the calculation. As I get from the code the formular for 2dAvgPooling you use is: (kernel_size + 1) * H_out * W_out * in_channels * out_channels
But if I consider the formular on the pytorch website I'd calculate them as follows:
For each kernel beeing applied on your input you have kernel_size * kernel_size - 1 additions plus one division plus one multiplication of your kernel_size inside the denominator. You apply out_channels kernels on in_channels input channels which result in a H_out * W_out output. This is the computation for all multiplications and additions. To be compliant with the regular understanding of MACs and FLOPs you devide by 2 as two MACs result in one FLOP. So finally I get the formular: FLOPs = (kernel_size * kernel_size - 1) * H_out * W_out * in_channels * out_channels / 2

The bias of a Linear layer is not taken into account for the FLOPs computation, as shown below:

Example:

import torch
import torch.nn as nn
from thop import profile

class Simple_FC_Net(nn.Module):
    def __init__(self, bias=True):
        super(Simple_FC_Net, self).__init__()
        self.fc = nn.Linear(in_features=17, out_features=49, bias=bias)

    def forward(self, x):
        x = self.fc(x)
        return x

input_simple_fc = torch.randn(1, 17)

# Model without bias
print("Model without bias:")
model = Simple_FC_Net(bias=False)
flops, params = profile(model, inputs=(input_simple_fc,))
print("Flops: {}, Params: {}".format(flops, params))

# Model with bias
print("Model with bias:")
model = Simple_FC_Net(bias=True)
flops, params = profile(model, inputs=(input_simple_fc,))
print("Flops: {}, Params: {}".format(flops, params))

Output:

Model without bias:
Register FLOP counter for module Linear(in_features=17, out_features=49, bias=False)
Flops: 1617.0, Params: 833.0

Model with bias:
Register FLOP counter for module Linear(in_features=17, out_features=49, bias=True)
Flops: 1617.0, Params: 882.0

Hi, the formula from wiki is as:

Your implementation is:
total_ops = y.nelement() * 13 # 6 muls + 7 adds
I can only count 5 for adds, could you please explain where 7 adds come from?
Thanks.

like the title

Firstly, I sincerely thank you for THOP, a much needed product for the community. However, I would like to know how to find the FLOPs when nodes are dropped. A followup question would be, if we freeze the layers, the forward propagation will be executed but weights wont be updated during the back propagation. How to calculate FLOPs when layers are frozen.

First of all, thanks to the author's selfless dedication, your code is well written!

Now 3D convolution is widely used. Have you considered supporting 3D convolution?

My model has two inputs. How can I use the code for a multiple-input model? Thanks.

Hi! I want to install this file offline through pip install thop. But it appear errors.
Environment:
unbuntu
python 3.6
torch 0.4.1

Thanks!

Hi!When my convolution kernel value is only 1 and -1 or when my input value is only 1 and -1, will the GFLOPS of the convolution operation be small? Thanks!

Thanks for your open source work，it's great！
I'm wondering how your bn layer flops is calculated，because your calculation result is not the same as the bn layer of ptflop package.
Also，compared to ptflops, I find you are not having to compute the relu layer flops.......

Hi

currently no counting method for LSTM layer! Please put it on road map.

thanks

Hi, first of all, thank you for your code. It really helps my work.
I am curious about the flops for depthwise convolutional layer, which should be different from the standard conv.
But it seems there is no implementation for such layer in your code. Could you please add support for it?
Thanks!

Thanks!

When I measure FLOPs of a Convolution layer with 3x3 kernel, 128 output channels, (1, 1, 128, 128) input data, "VALID" padding, thop and Tensorflow give me different results.

Tensorflow: 38612737
Torch: 20321280.0

Here is my code:

graph = tf.Graph()
with graph.as_default():
  print("* Initialize network")
  v = tf.keras.layers.Conv2D(128, (3, 3), padding="VALID", data_format="channels_first")(tf.placeholder(tf.float32, (1, 1, 128, 128)))
  print("Tensorflow:", tf.profiler.profile(graph, options=tf.profiler.ProfileOptionBuilder.float_operation()).total_float_ops)

print("Torch:",  thop.profile(torch.nn.Conv2d(1, 128, (3, 3)), inputs=(torch.zeros((1, 1, 128, 128)),), verbose=False)[0])

Why?

I use thop to calculate params, flops of vgg16, the code below:

The result is:
15.607G 138.358M

but in your readme.md,

File "/home/jhy/mf_py3.5/pytorch-OpCounter-master/thop/count_hooks.py", line 14, in count_convNd
kernel_ops = m.weight.size()[2:].numel()
AttributeError: 'torch.Size' object has no attribute 'numel'

When I used pip install thop, I got

Could not find a version that satisfies the requirement thop (from versions: )

When I installed thop from source and used it to resnet50 model, I got the following error message:

AttributeError Traceback (most recent call last)
in ()
----> 1 flops, params = profile(model, input_size=(1, 3, 224,224))

in profile(model, input_size, custom_ops, device)
73 x = torch.zeros(input_size).to(device)
74 with torch.no_grad():
---> 75 model(x)
76
77 total_ops = 0

/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py in call(self, *input, **kwargs)
475 result = self._slow_forward(*input, **kwargs)
476 else:
--> 477 result = self.forward(*input, **kwargs)
478 for hook in self._forward_hooks.values():
479 hook_result = hook(self, input, result)

/anaconda3/envs/pytorch/lib/python3.6/site-packages/torchvision/models/resnet.py in forward(self, x)
137
138 def forward(self, x):
--> 139 x = self.conv1(x)
140 x = self.bn1(x)
141 x = self.relu(x)

/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py in call(self, *input, **kwargs)
477 result = self.forward(*input, **kwargs)
478 for hook in self._forward_hooks.values():
--> 479 hook_result = hook(self, input, result)
480 if hook_result is not None:
481 raise RuntimeError(

in count_convNd(m, x, y)
10 batch_size = x.size(0)
11
---> 12 kernel_ops = m.weight.size()[2:].numel()
13 bias_ops = 1 if m.bias is not None else 0
14 ops_per_element = kernel_ops + bias_ops

AttributeError: 'torch.Size' object has no attribute 'numel'

kernel = torch.Tensor([*(x[0].shape[2:])]) // torch.Tensor(list((m.output_size,))).squeeze()
^
SyntaxError: invalid syntax

I think the count_conv2d function is for MACC or Multiplications.
In this function, total_ops is calculated by K x K x Cin x Wout x Hout X Cout.
Isn't it for the MACC calculation?

First, nice work. But I found something wrong when calculating flops in torch.nn.Conv2d, in count_hooks.py file, the out_w = y.size(2) // m.stride[0] is wrong, it must be out_w = y.size(2). The same as out_h.
So the flops result is less than standard ones.

Beside the entire model's flops and params, is there anyway to output Flops and Params for each component of the network?

kernel_ops = m.weight.size()[2:].numel() # Kw x Kh
AttributeError: 'torch.Size' object has no attribute 'numel'

May I ask why do you disable all the maxpooling part of your codes?

Hi,
Thank you for this repo. when I use this repo to count flops and params on pytorch0.4.1. there is an error occur like:
File "~/site-packages/thop/count_hooks.py", line 20, in count_convNd
kernel_ops = m.weight.size()[2:].numel() # Kw x Kh
AttributeError: 'torch.Size' object has no attribute 'numel'

I only use conv2d so I change this line like:
kernel_ops = m.weight.size()[2]*m.weight.size()[3]

I think you can offer different pytorch-OpCounter Branch for different pytorch version if there is someone needs. Or remind others who are in need.

Anyway, thanks and BRs.

Hi, thank you so much for your awesome work! I run the following code:
from torchvision.models import resnet50 from thop import profile model = resnet50() flops, params = profile(model, input_size=(1, 3, 224,224)) print(flops, params)

But I get the output:
4142713856.0 25557032.0 ,
which is different from your results in the table of README. And I'am using pytorch 1.0.1.
Could you help me explain that? Thank you!

Hi, thank you for your work! The output is number of params, how to convert it to memory usage just like in README?

Hi, thanks for your great work. I found an error in the following codes:

Firstly thank you for your exellent job.
When I try to calculate the FLOPs of MixNet which uses swish as non-linearity. And swish equals to x*sigmoid(x). But there are no hooks for both sigmoid or swish. So would you consider to add hooks for sigmoid or swish?
I know it is a little complex since Sigmoid(x) = 1/(1+exp(x)). The problem is how to calculate the FLOPs of exponetial operation. Maybe we can use multinomial to appoximate it? According to Maclaurin Series, exp(x) = 1 + x + x^2/2! + x^3/3! + .. + x^n/n!. The real problem becomes how should I choose the length of the infinite sequence.
Any one can help?

Hello, when I run your code. it appears the error"No Module "utils" and "denselink_imagenet"", how to solve it .Thank you !

It seems that the OpCounter doesn't take into account the skip connection structure.

Hi,

The tool you developed is very helpful.

But I am a little confused about the unit used in your result.

For example, here Params(M) should be the number of parameters, not the size?

And how about FLOPs(G), 1x10^9 flops the total forward and backward for a model? But the result you got is quite different from the tensorflow profiler's result. So, I am a little confused.

It seems that the reuse of module is banned by THOP tools

However, it is not quite reasonable. The re-use of ReLU-type module is very common, it can make your codes neater and causes no harm. e.g.

class Model(nn.Module):
  def __init__(self):
    super().__init__()
    self.dropout = nn.Dropout(0.5)
    self.relu = nn.ReLU()
    self.lin1 = nn.Linear(4096, 4096)
    self.lin2 = nn.Linear(4096, 4096)
    self.lin3 = nn.Linear(4096, 4096)
  def forward(self, x):
    output = self.relu(self.lin1(x))
    output = self.relu(self.lin2(output))
    output = self.dropout(self.relu(self.lin3(output)))
    ....

Besides, I think the network legality check is not THOP's work to do. This kind of restriction is unnecessary and will cause troublesome Error raise as I can see.