Colab Specific: RuntimeError: inverse_cuda: For batch 0: U(17437184,17437184) is zero, singular U. about fastai_dev HOT 3 CLOSED

No worries, thanks for investigating the issue!

from fastai_dev.

The fix is done, should come out when Colab rolls the update. Otherwise I think we can do a manual update. Will update when it's ready

Finally resolved the issue today.

It turns out that the CMake for magma 2.5.1 added the line set(CUDA_SEPARABLE_COMPILATION ON), and doing separable compilation somehow triggers some deep nvcc interaction with CentOS7 / RHEL7 linkers and all of that doesn't go down very well.

After setting CUDA_SEPARABLE_COMPILATION to OFF, we can successfully build static binaries that work correctly on a K80.

We did not root cause this bug, which needs significant more time and effort, because of it's subtle nature. We do not intend to root-cause it for now, as the cost-benefit isn't very high.

The commit that fixes the issue is pytorch/builder@dd7edfd

The nightlies from tomorrow morning will carry the fix.

The fix will also naturally be part of v1.3.1, after which we will request an update Google Colab, which will fix the originally flagged bug report.

from fastai_dev.

This issue has been solved. Recommend requiring torch 1.3.1 as this contains the fix

from fastai_dev.