QuatE: GPU memory is not released per epoch about pykeen HOT 3 OPEN

Do you have any hints where the problem comes from and how to fix it? I took a closer look at the QuatEInteraction and realised that a buffer called table is created. Could the problem perhaps lie here?

I think this buffer should be unrelated (it is created only once, and pretty small, too).

How does self.model look like?

The default setting of QuatE (the full model configuration, not the interaction) uses a regularizer, and it seems as if you use a custom training loop, so my best guess would be that the regularization term keeps accumulating without being back-propagated; in this case, torch would not be able to release tensors from previous batches.

from pykeen.

Thanks for your answer and sorry for my late reply!

I use the default QuatE model from the pykeen library here

It looks like this

QuatE(
  (loss): BCEWithLogitsLoss()
  (interaction): QuatEInteraction()
  (entity_representations): ModuleList(
    (0): Embedding(
      (regularizer): LpRegularizer()
      (_embeddings): Embedding(3007, 400)
    )
  )
  (relation_representations): ModuleList(
    (0): Embedding(
      (regularizer): LpRegularizer()
      (_embeddings): Embedding(12, 400)
    )
  )
  (weight_regularizers): ModuleList()
)

Are you talking about the weight_regularizers or the LpRegularizer()?

from pykeen.

I was talking about the two LpRegularizer instances shown as entity_representations[0].regularizer and relation_representations[0].regularizer; weight_regularizers is just an empty list 🙂

You can either

• disable the two regularizers by providing

model = QuatE(
    ...,
    entity_regularizer=None,
    relation_regularizer=None,
)

• or collect the regularization term by calling model.collect_regularization_term; you can either ignore this term (but in this case I would suggest to use the previous option instead), or add this to the loss before calculating the gradients.

As background info:

• weight_regularizers are regularizers that calculate a regularization term on all weights (of soem tensor, e.g., the relation embedding matrix);
• in contrast, the LpRegularizers you see above only calculate terms from the "activated" embeddings, i.e., the rows of the embedding matrix which are used in the current batch (and thus may also receive a non-zero gradient).

from pykeen.