Problem in grads about tf-keras-vis HOT 4 CLOSED

Hi, @IlvaHou . Thank you for letting us know that.

It seems the cause of problem is in tf-keras-vis, but I could NOT reproduce the error.
Could you please submit the stack trace of the error and the code snippet that reproduce the problem?

Thanks!

from tf-keras-vis.

This is the error.
For the code it might be hard to reproduce because I load pretrained weights in the model, but here is something:

DEFINE MODEL

import tensorflow.keras.backend as K
from tensorflow.keras.layers import Input, concatenate, Dense, Flatten, Conv2D, MaxPooling2D, AveragePooling2D, BatchNormalization, Dropout
from tensorflow.keras.models import Model

#Clear session
K.clear_session()

#L2-regularization
L2_weight = 0.00004

#Image branche
#Block 1
img_input = Input(shape=(250,250,1))
x_img = Conv2D(32, kernel_size=(3,3), padding='valid', activation='relu', kernel_regularizer=tf.keras.regularizers.l2(L2_weight))(img_input)
x_img = BatchNormalization()(x_img)
x_img = Conv2D(32, kernel_size=(3,3), padding='valid', activation='relu', kernel_regularizer=tf.keras.regularizers.l2(L2_weight))(x_img)
x_img = BatchNormalization()(x_img)
x_img = Conv2D(64, kernel_size=(3,3), padding='same', activation='relu', kernel_regularizer=tf.keras.regularizers.l2(L2_weight))(x_img)
x_img = BatchNormalization()(x_img)
x_img = MaxPooling2D(pool_size=(3,3), strides=(2,2), padding='valid')(x_img)

#Block 2
x_img = Conv2D(80, kernel_size=(1,1), padding='valid', activation='relu', kernel_regularizer=tf.keras.regularizers.l2(L2_weight))(x_img)
x_img = BatchNormalization()(x_img)
x_img = Conv2D(192, kernel_size=(3,3), padding='valid', activation='relu', kernel_regularizer=tf.keras.regularizers.l2(L2_weight))(x_img)
x_img = BatchNormalization()(x_img)
x_img = MaxPooling2D(pool_size=(3,3), strides=(2,2), padding='valid')(x_img)

#Block 3
x_img = Conv2D(512, kernel_size=(3,3), padding='valid', activation='relu', kernel_regularizer=tf.keras.regularizers.l2(L2_weight))(x_img)
x_img = BatchNormalization()(x_img)
x_img = MaxPooling2D(pool_size=(1,1), strides=(2,2), padding='valid')(x_img)

#Block 4
x_img = Conv2D(128, kernel_size=(3,3), padding='valid', activation='relu', kernel_regularizer=tf.keras.regularizers.l2(L2_weight))(x_img)
x_img = BatchNormalization()(x_img)
x_img = MaxPooling2D(pool_size=(1,1), strides=(2,2), padding='valid')(x_img)

#Block 5
x_img = Conv2D(256, kernel_size=(3,3), padding='valid', activation='relu', kernel_regularizer=tf.keras.regularizers.l2(L2_weight))(x_img)
x_img = BatchNormalization()(x_img)
x_img = Conv2D(256, kernel_size=(5,5), padding='valid', activation='relu', kernel_regularizer=tf.keras.regularizers.l2(L2_weight))(x_img)
x_img = BatchNormalization()(x_img)
x_img = AveragePooling2D(pool_size=(8,8), padding='valid')(x_img)
img_output = Flatten()(x_img)

#Gender branch
gen_input = Input(shape=(1,))
gen_output = Dense(16, activation='relu', kernel_regularizer=tf.keras.regularizers.l2(L2_weight))(gen_input)

#Concatenate
x_conc = concatenate([img_output, gen_output])
out1 = Dense(128, activation='relu', kernel_regularizer=tf.keras.regularizers.l2(L2_weight))(x_conc)
out1 = Dropout(0.5)(out1)
out1 = Dense(64, activation='relu', kernel_regularizer=tf.keras.regularizers.l2(L2_weight))(out1)
out1 = Dropout(0.5)(out1)
out = Dense(1, activation='linear')(out1)

model = Model([img_input, gen_input],out)

SHOW INPUT INSTANCE
An example input X looks like this:

[array([[[[1. ],
[1. ],
[1. ],
...,
[0.227451 ],
[0.227451 ],
[0.38431376]],

    [[0.22352943],
     [0.18039216],
     [0.16862746],
     ...,
     [0.227451  ],
     [0.21960786],
     [0.4156863 ]],

    [[0.2392157 ],
     [0.18431373],
     [0.17254902],
     ...,
     [0.21568629],
     [0.23529413],
     [0.4156863 ]],

    ...,

    [[0.61960787],
     [0.18823531],
     [0.16470589],
     ...,
     [0.1764706 ],
     [0.18431373],
     [0.22352943]],

    [[0.5137255 ],
     [0.20000002],
     [0.16862746],
     ...,
     [0.19215688],
     [0.20000002],
     [0.2392157 ]],

    [[0.54509807],
     [0.21960786],
     [0.20784315],
     ...,
     [0.22352943],
     [0.22352943],
     [0.23529413]]]], dtype=float32), array([1])]

TRY GRADCAMPLUSPLUS
Like mentioned above, I tried using GradCAMPlusPLus using the following code:

from tf_keras_vis.gradcam import GradcamPlusPlus

def loss(output):
return (output-168.93)

gradcam = GradcamPlusPlus(model)
cam = gradcam(loss, X, penultimate_layer=21)

Where for convenience I put down the ground truth value in the loss function manually and also the penultimate_layer.

One big side-note is that I tried this using TensorFlow version 1.14 (because that is with which I wrote the architecture of my model). And I did not receive any errors complaining about version, so I pursued. But this might become a bottleneck somewhere.

Thank you for checking!

from tf-keras-vis.

I found a way to achieve the result that I want without the use of this library. However, I think I made a custom version of regression activation mapping (as described in this paper: https://arxiv.org/pdf/1703.10757.pdf), instead of gradient activation mapping:

Get output of final convolutional layer

before_gap_layer = model.get_layer('activation_7')
before_gap_layer_output = Model(inputs=model.input, outputs=before_gap_layer.output)
before_gap_output = before_gap_layer_output.predict(X) # shape is (1,31,31,256)

Get weights of first dense layer

first_dense_layer = model.get_layer('dense_1')
first_dense_layer_weights = first_dense_layer.get_weights()[0]
first_dense_layer_weights_image = first_dense_layer_weights[0:256,:] # since this layer is based on concatenation of image information and gender information, we are only interested in the first 256 weights coming from the image branche # shape is (256,64)

Get weights of second dense layer

second_dense_layer = model.get_layer('dense_2')
second_dense_layer_weights = second_dense_layer.get_weights()[0] # shape is (64,1)

Combine weights to find attribution of convolutional layer to final prediction

total_weights = np.dot(first_dense_layer_weights_image, second_dense_layer_weights) # shape is (256,1)

Apply weights to get weighted sum of convolutional output as RAM

before_gap_layer_weighted = before_gap_output # initialize
for i, val in enumerate(total_weights):
before_gap_layer_weighted[: ,: ,: ,i] = np.multiply(before_gap_output[: ,: ,: ,i],val[0]) # shape is (1, 31, 31, 256)
ram = np.sum(before_gap_layer_weighted[0], axis=2)

Rescale and overlay to original image

from scipy.ndimage import zoom

ram = zoom(ram, (16.12, 16.12), order=1) # round(31x16.12)=500 is shape of original image
fix, ax = plt.subplots()
ax.imshow(X[0][0], alpha=0.5)
ax.imshow(ram, cmap='jet', alpha=0.5)

plt.show()

Gives an image like this:

(Image is a bit dark, but that has to do with the original image, not with this method)

But still if you know how to incorporate the gradients to create a grad-RAM for this model, please let me know!

from tf-keras-vis.

Hi, @IlvaHou . I'm sorry for the late reply.

One big side-note is that I tried this using TensorFlow version 1.14 (because that is with which I wrote the architecture of my model).

I regret that I have to say that tf-keras-vis is NOT supported Tensorflow 1.x, but only Tensorflow 2.x.
The cause of the error is that.

tf-keras-vis has been derived from keras-vis.
I've developed tf-keras-vis because it was difficult to improve and maintain keras-vis to support TF2.x.
So, when you use TF1.x, you have to select keras-vis.

I found a way to achieve the result that I want without the use of this library.

Thank you for letting us the interesting paper!
I'm going to read it.

I will close this issue, but if you have any question, please feel free to reopen this.
Thanks!

from tf-keras-vis.