Long delays on Titan V about benchmarks HOT 4 CLOSED

From the thread you linked, txbob said

If it is compiled against CUDA 8 (you would either have to compile from sources or pull a special wheel: tensorflow/tensorflow#14244 to get TF 1.4 compiled against CUDA 9) then you may be experiencing a substantial JIT-compilation delay. It's not unheard of for JIT to add minutes of start-up time to an application that uses many libraries (like TF). The best practice is to build TF from sources and make sure to specify the compute capability of the GPUs you are going to use:

Recent versions tf-nightly-gpu is compiled for CUDA 9 and compute capability 7.0 which is what the Titan V uses, so either build from source or use a recently tf-nightly-gpu version.

from benchmarks.

During the delay execution is in a sess.run (warm up / benchmark_one_step), so I couldn't go deeper.

from benchmarks.

Thanks @reedwm, the solution which I have just recently found, is that cuDNN 7.0.3 is not enough for Titan V, the delays were gone with 7.0.5.

from benchmarks.

.. and for the mentioned FP16 error message, see this issue:
tensorflow/tensorflow#15521

from benchmarks.