GPGPU-sim with cuDNN about gpgpu-sim_distribution HOT 7 OPEN

Hi,
I did not met this issue. In my case, loading ptxinfo and extracting ptx files were going well.

Did you set CUDA_INSTALL_PATH? GPGPU-Sim may call the ptxas command based on the value.

from gpgpu-sim_distribution.

Hi,
I did not met this issue. In my case, loading ptxinfo and extracting ptx files were going well.

Did you set CUDA_INSTALL_PATH? GPGPU-Sim may call the ptxas command based on the value.

Thank you. I have fixed the ptxas problem. (Although I do not know how did it get fixed by only switching a computer. )

Have you tried the MNIST example in https://github.com/gpgpu-sim/gpgpu-sim_simulations? Seems the developers use this app for testing... But finally, I got a PTX not implementation error.

from gpgpu-sim_distribution.

As you commented, I tried gpgpu-sim/gpgpu-sim_simulations.
When I run the mnistCUDNN, ptx extraction and some implicit_convolve_sgemm call work well. I could not find PTX not implementation error.
However, I've met a deadlock and the simulator has finally been dead as following message.

GPGPU-Sim uArch: Shader 32 bind to kernel 1 '_ZN5cudnn6detail23implicit_convolve_sgemmIffLi128ELi5ELi5ELi3ELi3ELi3ELi1ELb1ELb0ELb1EEEviiiPKT_iPT0_PS2_18kernel_conv_paramsiffiS6_S6_ii'
  <CTA alloc> : sm_idx=32 sid=32 max_cta_per_sm=16
GPGPU-Sim uArch: Shader 34 bind to kernel 1 '_ZN5cudnn6detail23implicit_convolve_sgemmIffLi128ELi5ELi5ELi3ELi3ELi3ELi1ELb1ELb0ELb1EEEviiiPKT_iPT0_PS2_18kernel_conv_paramsiffiS6_S6_ii'
  <CTA alloc> : sm_idx=34 sid=34 max_cta_per_sm=16
GPGPU-Sim uArch: ERROR ** deadlock detected: last writeback core 34 @ gpu_sim_cycle 12492 (+ gpu_tot_sim_cycle 4294867296) (87508 cycles ago)
GPGPU-Sim uArch: DEADLOCK  shader cores no longer committing instructions [core(# threads)]:
GPGPU-Sim uArch: DEADLOCK  0(64) 1(0) 2(64) 3(0) 4(64) 5(0) 6(64) 7(0)  + others ...  + others ...  + others ...  + others ...  + others ...  + others ...  + others ...  + others ...  + others ...  + others ...  + others ...  + others ...  + others ...  + others ...
Re-run the simulator in gdb and use debug routines in .gdbinit to debug this
Aborted (core dumped)

from gpgpu-sim_distribution.

As you commented, I tried gpgpu-sim/gpgpu-sim_simulations.
When I run the mnistCUDNN, ptx extraction and some implicit_convolve_sgemm call work well. I could not find PTX not implementation error.
However, I've met a deadlock and the simulator has finally been dead as following message.

GPGPU-Sim uArch: Shader 32 bind to kernel 1 '_ZN5cudnn6detail23implicit_convolve_sgemmIffLi128ELi5ELi5ELi3ELi3ELi3ELi1ELb1ELb0ELb1EEEviiiPKT_iPT0_PS2_18kernel_conv_paramsiffiS6_S6_ii'
  <CTA alloc> : sm_idx=32 sid=32 max_cta_per_sm=16
GPGPU-Sim uArch: Shader 34 bind to kernel 1 '_ZN5cudnn6detail23implicit_convolve_sgemmIffLi128ELi5ELi5ELi3ELi3ELi3ELi1ELb1ELb0ELb1EEEviiiPKT_iPT0_PS2_18kernel_conv_paramsiffiS6_S6_ii'
  <CTA alloc> : sm_idx=34 sid=34 max_cta_per_sm=16
GPGPU-Sim uArch: ERROR ** deadlock detected: last writeback core 34 @ gpu_sim_cycle 12492 (+ gpu_tot_sim_cycle 4294867296) (87508 cycles ago)
GPGPU-Sim uArch: DEADLOCK  shader cores no longer committing instructions [core(# threads)]:
GPGPU-Sim uArch: DEADLOCK  0(64) 1(0) 2(64) 3(0) 4(64) 5(0) 6(64) 7(0)  + others ...  + others ...  + others ...  + others ...  + others ...  + others ...  + others ...  + others ...  + others ...  + others ...  + others ...  + others ...  + others ...  + others ...
Re-run the simulator in gdb and use debug routines in .gdbinit to debug this
Aborted (core dumped)

Which configuration files are you using? When I used TITANX (Maxwell), I also got a deadlock. Could you try TITANV config instead?

Seems in your log file, you also got "Warning: cannot find deviceFun" --- I guess the problem is gpgpusim failed to find some device functions. (may be they do not belong to cuDNN? will check later. )

Thank you.

from gpgpu-sim_distribution.

Hello, I've found that mnistCUDNN require both libcudnn and libcublas, but GPGPU-Sim support extracting PTX files from only one library(its path is defined at PYTORCH_BIN variable).

After I modified GPGPU-Sim so that it supports extracting PTX files from multiple libraries.

I uploaded my pull request(PR #116).

Although mnistCUDNN calls several kernels for fermi or maxwell architecture(ex. cudnn7maxwell4gemm) with volta architecture config file, anyway it works well.

from gpgpu-sim_distribution.

Was mnistCUDNN linked statically with libcudnn and libcublas? I believe that was a requirement in order to account for gpgpu-sim only being able to extract PTX files from 1 file. The Makefile in deval281shah's mnistCUDNN example should be passing flags to statically link those libraries.

from gpgpu-sim_distribution.

@RedCarrottt @bigwater
Hello,

when I test minist/main.py, the output GPGPU-Sim PTX: Parsing libcaffe2_ gpu.1.sm_ 61.ptx lasted a long time and hasn't stopped yet. Have you ever encountered this problem when running this? Is it a mistake or does it take so long to handle PTX itself.

Thanks.

from gpgpu-sim_distribution.