terminate called after throwing an instance of 'c10::Error' about catalyst-rl-tutorial HOT 8 OPEN

We got this error as well, when was using the torch Dataloader in combination with cuda. I checked my OOM, and there was sufficient memory to perform the training. Setting num=workers to 1 did not help. Training on CPU did not help either.

Eventually, I found out that this error was caused because some of the data files I was trying to import with Dataloader were corrupted. The corrupted files caused this error, because they were incompatible with the torch.to_Tensor() function. This was not mentioned in the initial error message, I found it by setting the num_workers to 0. The error message then changed and included 'CORRUPTED file'. After removing the corrupted data files, everything ran smoothly with my initial num_workers=4.

from catalyst-rl-tutorial.

Hello @geekzyn! You are not using GPU, but you specify CUDA_VISIBLE_DEVICES=0 catalyst-rl run-trainer --config ./configs/config.yml. This looks fishy to me and a potential source of problems.

Either way, I would recommend you to get at least a single, modest GPU on your VM and try to rerun the code.

from catalyst-rl-tutorial.

I also encountered the same problem, but my hardware has a GPU.
Is there a solution?
For example, packages version issues, etc.
Environment: Ubuntu 18.04 LTS (with GPU / Cuda 10.0)

The Trainer Main Error Message:

================================================================================
Something go wrong with trajectory:
'NoneType' object has no attribute 'get'
None

terminate called after throwing an instance of 'c10::Error'
what(): HIP error: hipErrorNoDevice

from catalyst-rl-tutorial.

I have 2 GPU! but I have same problem

from catalyst-rl-tutorial.

Hello @geekzyn! You are not using GPU, but you specify CUDA_VISIBLE_DEVICES=0 catalyst-rl run-trainer --config ./configs/config.yml. This looks fishy to me and a potential source of problems.

Either way, I would recommend you to get at least a single, modest GPU on your VM and try to rerun the code.

Hi @dtransposed . I followed the toturial and encountered the same problem. And I found the reason was that in 'pip install -r requirements.txt' the pytorch version for ROCm was installed. But my gpu is Intel HD630 which is not supported by pytorch and is not an AMD card.

I first tried to run with CUDA_VISIBLE_DEVICES=“” ( also tried =-1) to use CPU only. But it seemed that the trainer part cannot run with CPU.(?) So I tired to uninstall pytorch in the virtualenv and then install torch==1.3.1+cpu. I also tried to modified the requirements.txt to install torch==1.3.1+cpu directly when running run-training.sh. But they did not work either.

My question is that is there any way to run the training demo sucessfully in my device (Intel HD630)(using only CPU or some other ways).

Besides, I have another problem. The trainer thread creates a vrep window which does not display anything. I suppose it is not normal right?

Thank you for your time. And I'm looking forward to your reply!

from catalyst-rl-tutorial.

0%| | 0/15721 [00:00<?, ?it/s]
terminate called after throwing an instance of 'c10::CUDAError'
what(): CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Exception raised from createEvent at ../aten/src/ATen/cuda/CUDAEvent.h:166 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7fc79731f612 in /data/nouman/.conda/envs/dcas22_task5/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: + 0xea8e4a (0x7fc7986dfe4a in /data/nouman/.conda/envs/dcas22_task5/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #2: + 0x339888 (0x7fc7e0ba7888 in /data/nouman/.conda/envs/dcas22_task5/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #3: c10::TensorImpl::release_resources() + 0x175 (0x7fc797304295 in /data/nouman/.conda/envs/dcas22_task5/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #4: + 0x214e5d (0x7fc7e0a82e5d in /data/nouman/.conda/envs/dcas22_task5/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #5: + 0x541618 (0x7fc7e0daf618 in /data/nouman/.conda/envs/dcas22_task5/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #6: THPVariable_subclass_dealloc(_object*) + 0x2b2 (0x7fc7e0daf912 in /data/nouman/.conda/envs/dcas22_task5/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #7: + 0x122adb (0x564df9970adb in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #8: + 0x1304ba (0x564df997e4ba in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #9: + 0x22082f (0x564df9a6e82f in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #10: _PyEval_EvalFrameDefault + 0x586d (0x564df998851d in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #11: + 0x152f2c (0x564df99a0f2c in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #12: + 0x1eae14 (0x564df9a38e14 in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #13: _PyEval_EvalFrameDefault + 0x5edd (0x564df9988b8d in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #14: _PyFunction_Vectorcall + 0x6f (0x564df9993c2f in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #15: + 0x1eae14 (0x564df9a38e14 in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #16: _PyEval_EvalFrameDefault + 0x654d (0x564df99891fd in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #17: _PyFunction_Vectorcall + 0x6f (0x564df9993c2f in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #18: + 0x1eae14 (0x564df9a38e14 in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #19: _PyEval_EvalFrameDefault + 0x654d (0x564df99891fd in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #20: _PyFunction_Vectorcall + 0x6f (0x564df9993c2f in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #21: + 0x1eae14 (0x564df9a38e14 in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #22: _PyEval_EvalFrameDefault + 0x5edd (0x564df9988b8d in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #23: + 0x152f2c (0x564df99a0f2c in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #24: + 0x1eae14 (0x564df9a38e14 in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #25: _PyEval_EvalFrameDefault + 0x5edd (0x564df9988b8d in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #26: _PyFunction_Vectorcall + 0x6f (0x564df9993c2f in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #27: + 0x1eae14 (0x564df9a38e14 in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #28: _PyEval_EvalFrameDefault + 0x654d (0x564df99891fd in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #29: _PyFunction_Vectorcall + 0x6f (0x564df9993c2f in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #30: + 0x1eae14 (0x564df9a38e14 in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #31: _PyEval_EvalFrameDefault + 0x654d (0x564df99891fd in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #32: _PyFunction_Vectorcall + 0x6f (0x564df9993c2f in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #33: + 0x1eae14 (0x564df9a38e14 in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #34: _PyEval_EvalFrameDefault + 0x5edd (0x564df9988b8d in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #35: _PyFunction_Vectorcall + 0x6f (0x564df9993c2f in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #36: + 0x1eae14 (0x564df9a38e14 in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #37: _PyEval_EvalFrameDefault + 0x5edd (0x564df9988b8d in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #38: _PyFunction_Vectorcall + 0x6f (0x564df9993c2f in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #39: + 0x1eae14 (0x564df9a38e14 in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #40: _PyEval_EvalFrameDefault + 0x654d (0x564df99891fd in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #41: + 0x1eac62 (0x564df9a38c62 in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #42: PyEval_EvalCode + 0x87 (0x564df9a38ba7 in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #43: + 0x1f234f (0x564df9a4034f in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #44: + 0x145e21 (0x564df9993e21 in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #45: + 0x1eae14 (0x564df9a38e14 in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #46: _PyEval_EvalFrameDefault + 0x654d (0x564df99891fd in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #47: _PyFunction_Vectorcall + 0x6f (0x564df9993c2f in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #48: + 0x1eae14 (0x564df9a38e14 in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #49: _PyEval_EvalFrameDefault + 0x654d (0x564df99891fd in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #50: _PyFunction_Vectorcall + 0x6f (0x564df9993c2f in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #51: + 0x1eae14 (0x564df9a38e14 in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #52: _PyEval_EvalFrameDefault + 0x5edd (0x564df9988b8d in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #53: _PyFunction_Vectorcall + 0x6f (0x564df9993c2f in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #54: + 0x1eae14 (0x564df9a38e14 in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #55: _PyEval_EvalFrameDefault + 0x5edd (0x564df9988b8d in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #56: _PyFunction_Vectorcall + 0x6f (0x564df9993c2f in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #57: _PyEval_EvalFrameDefault + 0x306 (0x564df9982fb6 in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #58: _PyFunction_Vectorcall + 0x6f (0x564df9993c2f in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #59: _PyEval_EvalFrameDefault + 0x49d9 (0x564df9987689 in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #60: + 0x1eac62 (0x564df9a38c62 in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #61: PyEval_EvalCode + 0x87 (0x564df9a38ba7 in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #62: + 0x1f234f (0x564df9a4034f in /data/nouman/.conda/envs/dcas22_task5/bin/python)
frame #63: + 0x145e21 (0x564df9993e21 in /data/nouman/.conda/envs/dcas22_task5/bin/python)

from catalyst-rl-tutorial.

i have gpu on my VM and done with extraction correctly but the training have this can anyone please help how to tackle this issue?

from catalyst-rl-tutorial.

i have gpu on my VM and done with extraction correctly but the training have this can anyone please help how to tackle this issue?

hello,I have one gpu and i have same problem,have you solved it? thanks

from catalyst-rl-tutorial.