<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clip

<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/us

Even if the above problem can be solved by rebasing to <a class="commit-link" data-hov

training fail with the following error about glat HOT 4 OPEN

flc777 commented on August 30, 2024

training fail with the following error

from glat.

Comments (4)

gyin94 commented on August 30, 2024

@FLC777 is there any suggested CUDA and pytorch Version to reproduce? Thanks.

from glat.

gyin94 commented on August 30, 2024

with CUDA_LAUNCH_BLOCKING=1 for debugging, the error came from the following one.

GLAT/glat_plugins/criterions/glat_loss.py

Line 63 in 6929c10

losses = F.nll_loss(logits, targets.to(logits.device), reduction="none")

/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:59: ClassNLLCriterion_updateOutput_no_reduce_kernel: block: [0,0,0], thread: [31,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.

RuntimeError: CUDA error: device-side assert triggered

The same dataset can be run with levenshtein_transformer without any issue.

from glat.

gyin94 commented on August 30, 2024

problem resolved by using the latest fairseq commit for pytorch 1.8.1+ facebookresearch/fairseq@9549e7f

from glat.

gyin94 commented on August 30, 2024

Even if the above problem can be solved by rebasing to facebookresearch/fairseq@9549e7f with pytorch 1.8.1+, another problem would come up. The training loss would be terrible (seemed to be reset to initial parameters) and won't change from the second epoch, (note the first epoch worked as expected). Is there any suggestion or other way to solve this issue? cc @FLC777

from glat.

Recommend Projects