Hi! I run into an error when trying to train a model.
My PyTorch version is 0.2.0_4
, and the cuda version is 8.0
.
Can you give me some insights about what's happening? Here is the trace:
(paravec) [zhangsheng@bbox1 paragraphvec]$ python train.py start --data_file_name 'example.csv' --num_epochs 100 --batch_size 32 --context_size 4 --num_noise_words 5 --vec_dim 150 --lr 1e-4
Dataset comprised of 4 documents.
Vocabulary size is 109.
Training started.
Epoch 1 - 20%/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [64,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [65,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [66,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [67,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [68,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [69,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [70,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [71,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [72,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [73,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [74,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [75,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [76,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [77,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [78,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [79,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [80,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [81,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [82,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [83,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [84,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [85,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [86,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [87,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [88,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [89,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [90,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [91,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [92,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [93,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [94,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [95,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [96,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [97,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [98,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [99,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [100,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [101,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [102,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [103,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [104,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [105,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [106,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [107,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [108,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [109,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [110,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [111,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [112,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [113,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [114,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [115,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [116,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [117,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [118,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [119,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [120,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [121,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [122,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [123,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [124,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [125,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [126,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/THCTensorIndex.cu:378: long calculateOffset(IndexType, LinearIndexCalcData<IndexType, Dims>) [with IndexType = unsigned int, Dims = 2U]: block: [9,0,0], thread: [127,0,0] Assertion `indexAtDim < data.baseSizes[dim]` failed.
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/generated/../THCReduceAll.cuh line=334 error=59 : device-side assert triggered
Traceback (most recent call last):
File "train.py", line 195, in <module>
fire.Fire()
File "/home/zhangsheng/anaconda2/envs/paravec/lib/python3.6/site-packages/fire/core.py", line 127, in Fire
component_trace = _Fire(component, args, context, name)
File "/home/zhangsheng/anaconda2/envs/paravec/lib/python3.6/site-packages/fire/core.py", line 366, in _Fire
component, remaining_args)
File "/home/zhangsheng/anaconda2/envs/paravec/lib/python3.6/site-packages/fire/core.py", line 542, in _CallCallable
result = fn(*varargs, **kwargs)
File "train.py", line 95, in start
save_all)
File "train.py", line 143, in _run
x = cost_func.forward(x)
File "/export/ssd/sheng/projects/paragraph-vectors/paragraphvec/loss.py", line 26, in forward
+ torch.sum(self._log_sigmoid(-scores[:, 1:]), dim=1) / k
File "/home/zhangsheng/anaconda2/envs/paravec/lib/python3.6/site-packages/torch/autograd/variable.py", line 476, in sum
return Sum.apply(self, dim, keepdim)
File "/home/zhangsheng/anaconda2/envs/paravec/lib/python3.6/site-packages/torch/autograd/_functions/reduce.py", line 16, in forward
return input.new((input.sum(),))
RuntimeError: cuda runtime error (59) : device-side assert triggered at /opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/generated/../THCReduceAll.cuh:334
terminate called without an active exception
Aborted (core dumped)