backprop question about graph_nn HOT 5 CLOSED

Thanks for your question.
If you draw a computational graph, where each node is a layer, nonlineary or another operation like gating, and where edges will show directions of forward and backward pass, you should be able to see that without the gating (or another operation) the projection operation in Graph U-Net becomes disconnected from the rest of the computational graph. Therefore, during backprop projection does not receive any signal to update its parameters.
More about computational graphs can be found here:
https://medium.com/tebs-lab/deep-neural-networks-as-computational-graphs-867fcaa56c9

from graph_nn.

Try this one to visualize computational graphs: https://github.com/szagoruyko/pytorchviz

from graph_nn.

Thanks, it helps me a lot.

I realized that I could use a mask to avoid backprop problem as you did in this repo. I need to set those remained nodes mask 1 after pooling, and others mask 0, rather than just throwing them away.

But one thing confused me a lot. How can I define that mask in tensorflow. I mean, it could not be a tensor, as tensors could not be altered, and it seems that a variable is not suitable. Besides, it could not be generated from other results, as it is a binary mask matrix indicates which node remained.

In brief, I could get indices of nodes I need to keep, then how could I define a max_nodes shape binary mask in tensorflow which allows me to set the corresponding position 0/1 ?

from graph_nn.

I'm not good at tensorflow, but I'm sure this should be possible. I think the way to go is to generate mask based on projection scores y. You can try finding some threshold alpha for each graph, so that if you do mask = y > alpha, then you will get the correct ratio of pooled nodes (say, 80% of values in mask will be 1 and 20% will be 0).

Alternatively, consider using a similar but different pooling method from our paper: Boris Knyazev, Graham W. Taylor, Mohamed R. Amer. Understanding Attention and Generalization in Graph Neural Networks, https://arxiv.org/abs/1905.02850
The PyTorch code is available in https://github.com/bknyaz/graph_attention_pool
In that case the parameter of the pooling method is the threshold alpha https://github.com/bknyaz/graph_attention_pool/blob/master/attention_pooling.py#L161
So, it should be easier to implement in tensorflow. Plus this method has nicer properties as we discuss in the paper.

from graph_nn.

Yes, this does work, thanks for your suggestions. And I will absolutely read your paper for some new thoughts.

from graph_nn.