wielandbrendel / bag-of-local-features-models Goto Github PK

Wonderful work ! Can you share foolbox file ? Thanks！

Line #53 to #55 :I think you are trying to downsample, or your down sample failed, raise some bugs so you tried to 'crop' a part of the residual directly, and add it on the output, to make the dimension of residual and output equal, to solve the raising error, but I think this is not the appropriate way to 'add' residual to the output!

Line #78 nn.AvgPool(1, stride=1) will make no difference on the input tensor, so the output of this avgpool is the same as before. So interestingly, you add another avgpool operation in the BagNet's 'forward' method.... If you want the tensor to be the size you want after average pooling, you should just use nn.AdaptiveAvgPool().

And I really want to know how you train your BagNet. Is the training data the whole image or image patches?

Hello,

 I have found your work to be very fascinating, especially the potential for much better visualizations of what has been learned by the network.  I was playing around with the notebook you uploaded, but can't seem to get the visualizations to look like the ones that are shown in your paper.  Unfortunately, I don't know the index numbers of those specific images so I couldn't do an exact comparison.  However, the images I did go through didn't seem to be comparable at all in quality to the ones in the paper.  How would I reproduce the images you used as examples in your paper?  Thank you for publishing such great work here for everyone to learn from.

Which pytorch version is required? will 1.0.0 work?
Thanks!

What a wonderful and impressive work!

But I was a little confused about the detail. The following is my understanding, could you please point out whether it is right or not?

Bagnet is nothing different from resnet both in training and testing procedure except that some of the 3*3 convs are replaced with 1*1 convs for smaller receptive field. In your code, you feed a large image(224*224) to bagnet33 in training. Actually it is equivalent that we feed the same image in patches to bagnet33 which are generated by function generate_heatmap_pytorch (patchsize=33, stride=8) and average the logits of each patch.(is it right?)

But when I tried feeding the same image in patches and averaging the patches result in training, it seemed something was wrong and its prediction result was bad. I was wondering if it was something I didn't quite understood?

Thanks in advance and I can't help saying that this paper is extremely useful!

Wonderful work. I am curious about the effect of patch size on the performance. In the paper, only q=9,17,33 are tested. How about bigger size?

what version of keras & tensorflow are supported? i'm having no luck with keras 2.2.4 and either tensorflow 1.11.0 or 1.12.0.

import bagnets.keras
keras_model = bagnets.keras.bagnet16()

Traceback (most recent call last):
  File "/usr/local/anaconda2/envs/scratchy/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1628, in _create_c_op
    c_op = c_api.TF_FinishOperation(op_desc)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Shape must be rank 1 but is rank 0 for 'bn10.01234546713855944/cond/Reshape_4' (op: 'Reshape') with input shapes: [1,64,1,1], [].

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/anaconda2/envs/scratchy/lib/python3.6/site-packages/bagnets/keras.py", line 28, in bagnet16
    return load_model(model_path)
  File "/usr/local/anaconda2/envs/scratchy/lib/python3.6/site-packages/keras/engine/saving.py", line 419, in load_model
    model = _deserialize_model(f, custom_objects, compile)
  File "/usr/local/anaconda2/envs/scratchy/lib/python3.6/site-packages/keras/engine/saving.py", line 225, in _deserialize_model
    model = model_from_config(model_config, custom_objects=custom_objects)
  File "/usr/local/anaconda2/envs/scratchy/lib/python3.6/site-packages/keras/engine/saving.py", line 458, in model_from_config
    return deserialize(config, custom_objects=custom_objects)
  File "/usr/local/anaconda2/envs/scratchy/lib/python3.6/site-packages/keras/layers/__init__.py", line 55, in deserialize
    printable_module_name='layer')
  File "/usr/local/anaconda2/envs/scratchy/lib/python3.6/site-packages/keras/utils/generic_utils.py", line 145, in deserialize_keras_object
    list(custom_objects.items())))
  File "/usr/local/anaconda2/envs/scratchy/lib/python3.6/site-packages/keras/engine/network.py", line 1032, in from_config
    process_node(layer, node_data)
  File "/usr/local/anaconda2/envs/scratchy/lib/python3.6/site-packages/keras/engine/network.py", line 991, in process_node
    layer(unpack_singleton(input_tensors), **kwargs)
  File "/usr/local/anaconda2/envs/scratchy/lib/python3.6/site-packages/keras/engine/base_layer.py", line 457, in __call__
    output = self.call(inputs, **kwargs)
  File "/usr/local/anaconda2/envs/scratchy/lib/python3.6/site-packages/keras/layers/normalization.py", line 206, in call
    training=training)
  File "/usr/local/anaconda2/envs/scratchy/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 3123, in in_train_phase
    x = switch(training, x, alt)
  File "/usr/local/anaconda2/envs/scratchy/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 3058, in switch
    else_expression_fn)
  File "/usr/local/anaconda2/envs/scratchy/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
    return func(*args, **kwargs)
  File "/usr/local/anaconda2/envs/scratchy/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2097, in cond
    orig_res_f, res_f = context_f.BuildCondBranch(false_fn)
  File "/usr/local/anaconda2/envs/scratchy/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 1930, in BuildCondBranch
    original_result = fn()
  File "/usr/local/anaconda2/envs/scratchy/lib/python3.6/site-packages/keras/layers/normalization.py", line 167, in normalize_inference
    epsilon=self.epsilon)
  File "/usr/local/anaconda2/envs/scratchy/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 1908, in batch_normalization
    mean = tf.reshape(mean, (-1))
  File "/usr/local/anaconda2/envs/scratchy/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 6482, in reshape
    "Reshape", tensor=tensor, shape=shape, name=name)
  File "/usr/local/anaconda2/envs/scratchy/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/usr/local/anaconda2/envs/scratchy/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
    return func(*args, **kwargs)
  File "/usr/local/anaconda2/envs/scratchy/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3274, in create_op
    op_def=op_def)
  File "/usr/local/anaconda2/envs/scratchy/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1792, in __init__
    control_input_ops)
  File "/usr/local/anaconda2/envs/scratchy/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1631, in _create_c_op
    raise ValueError(str(e))
ValueError: Shape must be rank 1 but is rank 0 for 'bn10.01234546713855944/cond/Reshape_4' (op: 'Reshape') with input shapes: [1,64,1,1], [].

Hello @wielandbrendel. In the paper How Figure 5 which illstrates the examples of original and texturised images are created? I mean how did you use style transfer that lead to scrambbled images? Thanks

hi, I've recently read your paper, your thoughts were instructive and the visualization works look great, and I would like to discuss some of my thoughts about your work, which I'm not quite sure:

In my understanding, the reported results are obtained in such a way that:

1. a 33 receptive-field bagnet is trained on 224x224 full imagenet in original convnet-way
2. the trained bagnet is applied to patches splited from 224x224 imagenet pictures with padding, and if we have nxn patches, we get nxnx1000 heat maps and softmax(sum()) such heatmaps gives the 1x1000 cls token

If above is right, it is not that surprising that such architecture works, since the ability to extract features is learnt in step 1, though weaker than original resnets, but I think resnet-like DNNs enjoys good robustness. Did you test the step 1 trained net directly on full-size imagenet? Is there great difference between full-size and patch-based classification result using same bagnet trained in step 1? (or did you already reported but i missed it-.-) If the two results are similar, the conclusion might degrade to "DNN with less receptive field but same depth doesn't suffer from much decrease in terms of top-5 acc"

Thanks!

Hi！
Thank you for your reply for my question about input of Bagnets, I learned your code today, and I calucalate the receptive field of bagnet-9 by Fomoro AI . The results are as follows. The input size is 224×224, I ignore many layers (stride=1 and kernel size =1, it does not change the receptive field ). The receptive field is 9×9, but the output size is 27×27. It means the receptive fields of different neuron overlaps because 27×9=243>224. I think there's something wrong with my results. But I could not find this problem. I would appreciate it if you could give us some advice on how to solve this problem.

Wonderful idea and result.

Would you plan to release code of training? There are many details making reproduction difficult, such as batch size, data augmentation, patches selection and so on. I sincerely hope you could consider my request.

Thanks!

Thanks for providing the code!

I don't have any issue loading the models in pytorch (the code below),

import bagnets.pytorch
pytorch_model = bagnets.pytorch.bagnet17(pretrained=True)

But in keras the code below throws an error!

import bagnets.keras
keras_model = bagnets.keras.bagnet17()

Probably I'm doing something wrong.
I would appreciate your help.

The error is in the below:

keras_model = bagnets.keras.bagnet17()

Traceback (most recent call last):
File "", line 1, in
File "/home/kd/anaconda2/bin/py27_1/local/lib/python2.7/site-ackages/keras/engine/saving.py", line 419, in load_model
model = _deserialize_model(f, custom_objects, compile)
File "/home/kd/anaconda2/bin/py27_1/local/lib/python2.7/site-ackages/keras/engine/saving.py", line 225, in _deserialize_model
model = model_from_config(model_config, custom_objects=custom_objects)
File "/home/kd/anaconda2/bin/py27_1/local/lib/python2.7/site-ackages/keras/engine/saving.py", line 458, in model_from_config
return deserialize(config, custom_objects=custom_objects)
File "/home/kd/anaconda2/bin/py27_1/local/lib/python2.7/site-packages/keras/layers/init.py", line 55, in deserialize
printable_module_name='layer')
File "/home/kd/anaconda2/bin/py27_1/local/lib/python2.7/site-packages/keras/utils/generic_utils.py", line 145, in deserialize_keras_object
list(custom_objects.items())))
File "/home/kd/anaconda2/bin/py27_1/local/lib/python2.7/site-packages/keras/engine/network.py", line 1022, in from_config
process_layer(layer_data)
File "/home/kd/anaconda2/bin/py27_1/local/lib/python2.7/site-packages/keras/engine/network.py", line 1008, in process_layer
custom_objects=custom_objects)
File "/home/kd/anaconda2/bin/py27_1/local/lib/python2.7/site-packages/keras/layers/init.py", line 55, in deserialize
printable_module_name='layer')
File "/home/kd/anaconda2/bin/py27_1/local/lib/python2.7/site-packages/keras/utils/generic_utils.py", line 145, in deserialize_keras_object
list(custom_objects.items())))
File "/home/kd/anaconda2/bin/py27_1/local/lib/python2.7/site-packages/keras/layers/core.py", line 735, in from_config
function = func_load(config['function'], globs=globs)
File "/home/kd/anaconda2/bin/py27_1/local/lib/python2.7/site-packages/keras/utils/generic_utils.py", line 235, in func_load
code = marshal.loads(raw_code)
ValueError: bad marshal data (unknown type code)

I am trying to load the pre-trained Keras models using Tensorflow 2.1.0 (Python 3.6.9). I am getting the following err (I also tried tf 1.14 and tf 2.0, same result):

Traceback (most recent call last):
File "test_bagnet.py", line 40, in
model = bagnet17()
File "test_bagnet.py", line 28, in bagnet17
return load_model(model_path)
File "/home/hmueller/venv3/lib/python3.6/site-packages/tensorflow_core/python/keras/saving/save.py", line 146, in load_model
return hdf5_format.load_model_from_hdf5(filepath, custom_objects, compile)
File "/home/hmueller/venv3/lib/python3.6/site-packages/tensorflow_core/python/keras/saving/hdf5_format.py", line 168, in load_model_from_hdf5
custom_objects=custom_objects)
File "/home/hmueller/venv3/lib/python3.6/site-packages/tensorflow_core/python/keras/saving/model_config.py", line 55, in model_from_config
return deserialize(config, custom_objects=custom_objects)
File "/home/hmueller/venv3/lib/python3.6/site-packages/tensorflow_core/python/keras/layers/serialization.py", line 106, in deserialize
printable_module_name='layer')
File "/home/hmueller/venv3/lib/python3.6/site-packages/tensorflow_core/python/keras/utils/generic_utils.py", line 303, in deserialize_keras_object
list(custom_objects.items())))
File "/home/hmueller/venv3/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/network.py", line 937, in from_config
config, custom_objects)
File "/home/hmueller/venv3/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/network.py", line 1903, in reconstruct_from_config
process_node(layer, node_data)
File "/home/hmueller/venv3/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/network.py", line 1851, in process_node
output_tensors = layer(input_tensors, **kwargs)
File "/home/hmueller/venv3/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 748, in call
self._maybe_build(inputs)
File "/home/hmueller/venv3/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 2116, in _maybe_build
self.build(input_shapes)
File "/home/hmueller/venv3/lib/python3.6/site-packages/tensorflow_core/python/keras/layers/core.py", line 1101, in build
raise ValueError('The last dimension of the inputs to Dense '
ValueError: The last dimension of the inputs to Dense should be defined. Found None.

Is there a workaround? In issue #19 you mentioned loading the models in tensorflow successfully, which version did you use?

Hi, appreciate your work.
I'm wondering to train it from scratch, should I set batch_size to be one? since the number of patches takes the batch dimension.

Thanks

Hi,
Many thanks to having published the paper together with the code. By playing with your code, I come up with some very simple questions. You have BagNet-33 with 33X33 as the receptive field size. I am wondering how to obtain 33 exactly? The BagNet-33 only can take a minimal input image size of 41X41.

Hi, first of all, your work is so inspiring!
I want to consider how the receptive field is established in the network:

In BagNet-9:

The first convolutional layer uses a kernel size of 1x1. Since the stride is 1 by default, each output neuron in this layer sees a receptive field of size 1x1, centered around a single pixel in the input image.

The second convolutional layer uses a kernel size of 3x3 with a stride of 2. This means that each output neuron in this layer sees a receptive field of size 3x3, but with a stride of 2, the receptive fields are not directly adjacent. The receptive fields overlap but are not contiguous.

The third and fourth convolutional layers also use 3x3 kernels with a stride of 2. Similar to the second layer, the receptive fields of neurons in these layers overlap but are not directly adjacent.
Given the strides used in the convolutional layers, the 9x9 receptive field of BagNet-9 is not composed of directly adjacent pixels in the input image.

in your heatmaps visualization, you are dividing the input image into all 9x9 patches, assuming the 9x9 pixels are direct neighbors in the input image.
what am I missing?

Thanks in advance,
Bar

Wonderful job！I studied your paper these days, which is very enlightening to me. I want to use the bagnets in my research. But I am not very sure the input of the bagnets. For example, If I want to classify a 100*100 image, I should first divide the image into some smaller patches as the input of the bagnets, not the whole image, is it right?

Hello, I noticed that the model weights of Keras may be corrupted. I have tried different versions , but the issue persists. Has anyone else experienced this issue, or is it just me? Thank you for any help in advance!

Great work. It give me a lot insights. Can you please release the training code (Get the BagNet by myself)?

Hello,

First, thanks at lot for providing code with the article ! However, just a quick question, my understanding from reading the paper was that after the last convolutional step, you apply a linear classifier, sayDense(1000), from each image patch and then average those prediction over all patches.
In the code (here), it looks like you first AvgPool2d() then Dense(1000).
Even though the order of this operation commutes, I was wondering if there were a reason for that or if it's something I didn't quite understood ?

Thanks !

Hello,

This is really a great work! thanks a lot for making the codebase public. I was wondering if its possible to get individual embedding of each patch of size (q x q). From the pytorch model architecture, I was not sure how to do that.

Thanks!

Hi!
I was looking over the code to check how could I custom the model architecture to train a classification problem of mine, with two classes only. I could not find any compile or build or fit model in order to do this, am I right? Or am I looking something wrong?

Thanks in advance and I did not want to leave without saying that I've found the paper extremely useful and understandable. Really appreaciated .)

bagnet33 pytorch result on a image

[('Labrador retriever', 96.07062530517578),
('Saluki, gazelle hound', 3.3289875984191895),
('golden retriever', 0.33122718334198),
('bull mastiff', 0.11044301837682724),
('Great Dane', 0.04927573725581169)]

bagnet33 tensorflow result on the same image
('n02099712', 'Labrador_retriever', 28.739027)
('n02091831', 'Saluki', 25.376663
('n02099601', 'golden_retriever', 23.068756)
('n02108422', 'bull_mastiff', 21.97139)
('n02109047', 'Great_Dane', 21.164375)

These result were produced by resizing image to (224,224)
The difference persist on bagnet17 as well.
Is this difference expected?
The image used is attached

.

The heatmap in the example notebook is probably wrong since the model is not in evaluation mode and thus the Batchnorm in the model doesn't work properly.

https://github.com/wielandbrendel/bag-of-local-features-models/blob/master/bagnets/pytorch.py#L142
should be named bagnet17 instead according to Usage and the paper

This is really great work! I'm not sure I understand how you generated the heat maps shown in your paper. Is there any code available that demonstrates this?

I have found your work to be very fascinating, especially the potential for much better visualizations of what has been learned by the network. I was playing around with the notebook you uploaded, but can't seem to get the visualizations to look like the ones that are shown in your paper. Unfortunately, I don't know the index numbers of those specific images so I couldn't do an exact comparison. However, the images I did go through didn't seem to be comparable at all in quality to the ones in the paper. How would I reproduce the images you used as examples in your paper? Thank you for publishing such great work here for everyone to learn from.