Workaround for tokens with impossible states? ("problem found for token X" error) about skweak HOT 1 CLOSED

Hi Alfredo,
Thanks (and sorry for the late answer)!

Yes, you're perfectly right, once could certainly replace the negative infinity by a large negative number, and it's something we had actually tested in a previous version. However, we decided to go for the current solution for two reasons:

• the first reason is computational, since alternatives with a zero probability can be directly skipped, which can speed up inference by reducing the number of alternatives to consider ;
• the second reason relates to debugging: typically, if one reaches a state where no possible state value has a likelihood > 0, this signals a serious problem that should be fixed ASAP, as it is likely to create many other convergence problems down the line. It's something that happened to us quite a few times when testing various labelling functions, and we found it quite useful to get a clear error message thrown when that happens.

This being said, it's true that the current is quite cryptic, and we should do a better job at providing more informative cues about what's happening and what should be fixed.

from skweak.