Models breaking after several hours about df HOT 3 CLOSED

I've see this happen too with the original trainer as well. Not sure what is going on. Have had some luck with lowering the batch sizes. Looking into it. I had a Issue loaded on the main branch of faceswap, but I closed it as I thought I had fixed the issue, but no luck as of yet as I am still getting the issue at times.

from df.

@kellurian I've now found, though I am not completely certain about this, that if you rerun alignments on an alignments directory the likelihood you'll get the "red screen of death" decreases. It might be that in certain situations, if any images are missings in a directory and according to alignments.json those images are suppossed to be there, when the training process tries to get to those missing images, the process fails and ruins the model. I didn't check the code for exceptions - my programming abilities are fairly limited- so I don't know if these situations are in fact being handled properly, but up until now my anecdotal evidence seems to indicate it is helping.

from df.

@ And just as I type this the model breaks. Wow, incredible timing. It had been running for well over 24 hours, this is the longest I've had it work. I guess you should completely disregard what I just said. I honestly have no clue why this might be.

from df.