bayer-science-for-a-better-life / phc-gnn Goto Github PK

Hi

I would like to initialize phm_rule and weights in a way that the final weight matrix of PHM layers is initialized with normal(mean=0, std=0.01), could you kindly provide me with some suggestions on how this can be achieved? So which initialization I can use for phm_rules and weight variables.

thanks

General idea: When training is finished, the PHM-weight matrix does not have to be constructed by a sum of Kronecker products in every forward pass. One could check for self.train and then use a cached version, e.g. saved in self._cached_weight of the final weight matrix for the linear transformation.
Using torch.nn.functional.linear function with inserting the self._cached_weight parameter.

Advantage: speedup gain in inference, as trainable parameters for PHM-layer are not updated, when running inference, we could just the final weight matrix, which is constructed by the optimized contribution and sub-weight matrices {C_i}_{i=1}^{n} and {W_i}_{i=1}^{n} , respectively.

Hi，

when I run this code via bash run_script_pcba_phm4.sh

I got the following error：

Traceback (most recent call last):
File "train_pcba.py", line 634, in
main()
File "train_pcba.py", line 610, in main
ogb_bestEpoch_test_metrics, ogb_lastEpoch_test_metric, ogb_val_metrics = do_run(i, model, args,
File "train_pcba.py", line 327, in do_run
train_metrics = train(epoch=epoch, model=model, device=device, transform=transform,
File "train_pcba.py", line 174, in train
logits = model(data)
File "/root/miniconda3/envs/phc-gnn/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/ogb/phc-gnn-master/phc/hypercomplex/undirectional/models.py", line 243, in forward
x = self.compute_hidden_layer_embedding(conv=self.convs[i], norm=self.norms[i],
File "/ogb/phc-gnn-master/phc/hypercomplex/undirectional/models.py", line 208, in compute_hidden_layer_embedding
x = conv(x=tmp[0], edge_index=edge_index, edge_attr=edge_attr, size=size)
File "/root/miniconda3/envs/phc-gnn/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/ogb/phc-gnn-master/phc/hypercomplex/undirectional/messagepassing.py", line 519, in forward
return self.transform(x, edge_index, edge_attr, size)
File "/root/miniconda3/envs/phc-gnn/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/phc-gnn-master/phc/hypercomplex/undirectional/messagepassing.py", line 60, in forward
x = self.propagate(edge_index=edge_index, x=x, edge_attr=edge_attr, size=size)
File "/root/miniconda3/envs/phc-gnn/lib/python3.8/site-packages/torch_geometric/nn/conv/message_passing.py", line 236, in propagate
out = self.message(**msg_kwargs)
File "/ogb/phc-gnn-master/phc/hypercomplex/undirectional/messagepassing.py", line 74, in message
assert x_j.size(-1) == edge_attr.size(-1)
AssertionError

Then， I print the shape of both x_j and edge_attr，
x_j : [28302, 512]
edge_attr : [28302, 2048]

what caused this error?

Hi
I observe speed difference between the PHM layers and linear layers, while in the original paper they clarim the running time should be the same, could you kindly let me know if you might know a way I could make the running time faster? greatly appreciated.

Hi there,

I am using PHMLinear here [1], then I set all the seeds (torch/torch cuda/numpy/random) once before making a model, and then I use this layer inside the model, I am getting each time a different results, could you guide me please how I can fix this issue with this layer?

thank you

[1]

Hi
thanks for sharing your codes. I am a bit confused on finding the implementation of PHM-layer and how I can turn a linear layer to the more efficient one using the method in the paper to train the models, could you kindly guide me?
I found this line

Hi，

When I run the code via bash run_script_hiv_phm4.sh

I get the following error：

Traceback (most recent call last):
File "train_hiv.py", line 636, in
main()
File "train_hiv.py", line 569, in main
model = UPH_SC_ADD(phm_dim=phm_dim, learn_phm=learn_phm, phm_rule=phm_rule,
File "/ogb/phc-gnn-master/phc/hypercomplex/undirectional/models.py", line 134, in init
self.convs[i] = PHMMessagePassing(in_features=in_dim, out_features=out_dim, bias=bias,
File "/ogb/phc-gnn-master/phc/hypercomplex/undirectional/messagepassing.py", line 494, in init
self.transform = PHMGINEConvSoftmax(in_features, out_features, phm_dim, phm_rule, learn_phm,
File "/ogb/phc-gnn-master/phc/hypercomplex/undirectional/messagepassing.py", line 277, in init
self.transform = PHMMLP(in_features=in_features, out_features=out_features, phm_dim=phm_dim,
File "/phc-gnn-master/phc/hypercomplex/layers.py", line 324, in init
self.linear1 = PHMLinear(in_features=in_features, out_features=int(factor*out_features),
File "/ogb/phc-gnn-master/phc/hypercomplex/layers.py", line 231, in init
assert in_features % phm_dim == 0, f"Argument in_features={in_features} is not divisble be phm_dim{phm_dim}"
AssertionError: Argument in_features=50 is not divisble be phm_dim4

The default embedding dimension is 200. Does this cause this error?

Hi
I tried to test your kronecker product implementation and compare it with torch implementation, for the matrices of the dimension on the example you provided in #1 :

def kronecker_product1(a, b):
    siz1 = torch.Size(torch.tensor(a.shape[-2:]) * torch.tensor(b.shape[-2:]))
    res = a.unsqueeze(-1).unsqueeze(-3) * b.unsqueeze(-2).unsqueeze(-4)
    siz0 = res.shape[:-4]
    out = res.reshape(siz0 + siz1)
    return out

def kronecker_product2(a, b):
   return torch.kron(a, b)

Now lets test the two versions:

a = torch.rand((4, 4, 4))
b = torch.rand((4, 64, 32))
result1 = kronecker_product1(a, b)
print("result 1 ", result1.shape)
result2 = kronecker_product2(a, b)
print("result 2 ", result2.shape)

The results are not the same and even dimension-wise they differ:

result 1  torch.Size([4, 256, 128])
result 2  torch.Size([16, 256, 128])

Could you kindly assist me how I can use the torch.kron implementation and which modifications are needed?

thanks