Invalid PTX input on RTX 2080 Ti on Windows about pbrt-v4 HOT 17 CLOSED

The good news is that removing debugging symbols from the PTX generated for the optix kernels seems to fix the problem. The bad news is that I haven't been able to figure out how to do that for just the PTX build, so the diff below fixes it, but at the cost of not having debugging symbols in any of the other stuff built with the CUDA compiler, which is undesirable...

Heads up there also seems to be a lingering build dependency issue around this stuff. (e.g., if I run touch gpu/optix.cu it seems to only rebuild the .ptx file but not also run bin2c, etc, and update the pbrt executable...)

diff --git a/CMakeLists.txt b/CMakeLists.txt
index e7bd6c7..ca59d50 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -174,7 +174,7 @@ if (CMAKE_CUDA_COMPILER)
 
         set (CMAKE_CUDA_FLAGS "${CMAKE_CUDA_FLAGS} --std=c++17")
         if (CMAKE_BUILD_TYPE MATCHES Debug)
-          set (CMAKE_CUDA_FLAGS "${CMAKE_CUDA_FLAGS} --use_fast_math -G -g")
+          set (CMAKE_CUDA_FLAGS "${CMAKE_CUDA_FLAGS} --use_fast_math")
         else()
           set (CMAKE_CUDA_FLAGS "${CMAKE_CUDA_FLAGS} --use_fast_math -lineinfo -maxrregcount 128")
         endif ()
@@ -213,6 +213,8 @@ if (CMAKE_CUDA_COMPILER)
                 # disable "extern declaration... is treated as a static definition" warning
                 -Xcudafe=--display_error_number -Xcudafe=--diag_suppress=3089
                 )
+          set(CMAKE_CUDA_FLAGS_DEBUG "")
+
           # CUDA integration in Visual Studio seems broken as even if "Use
           # Host Preprocessor Definitions" is checked, the host preprocessor
           # definitions are still not used when compiling device code.

from pbrt-v4.

Okay, with the diff I posted above OptiX seems to now be happy in debug build with the PTX, but I then run in an illegal memory access in CUDA in GPUPathIntegrator::GenerateCameraRays<pbrt::HaltonSampler>(), however I will keep that for a separate issue.

from pbrt-v4.

I'm updating now to OptiX 7.20, (but leaving 7.10) and 456.71 drivers ( I have an NVidia GTX 1060.)
Mean while, without your last commit, I get this error:

pbrt version 4 (built Oct 22 2020 at 23:52:41)
Copyright (c)1998-2020 Matt Pharr, Wenzel Jakob, and Greg Humphreys.
The source code to pbrt (but *not* the book contents) is covered by the Apache 2.0 License.
See the file LICENSE.txt for the conditions of the license.
[ 3868.000 20201023.004908 D:/REPOS/bountypbrt4/src/pbrt/gpu/accel.cpp:620 ] FATAL OptiX call optixModuleCreateFromPTX(optixContext, &moduleCompileOptions, &pipelineCompileOptions, ptxCode.c_str(), ptxCode.size(), log, &logSize, &optixModule) failed with code 7200: "Invalid PTX input"
Logs: COMPILE ERROR: Invalid PTX input: ptx2llvm-module-001: error: Failed to parse input PTX string
ptx2llvm-module-001, line 9; warning : Unsupported .version 7.0; current version is '6.4'
ptx2llvm-module-001, line 82837; fatal   : Parsing error near 'X': syntax error
Cannot parse input PTX string


(unknown                                 )      0x0000137000001010 - ░�
(unknown                                 )      0x0000137000001010 - ░�
(unknown                                 )      0x0000137000001010 - ░�
(unknown                                 )      0x0000137000001010 - ░�
(unknown                                 )      0x0000137000001010 - ░�
(unknown                                 )      0x0000137000001010 - ░�
(unknown                                 )      0x0000137000001010 - ░�
(unknown                                 )      0x0000137000001010 - ░�
(unknown                                 )      0x0000137000001010 - ░�
(unknown                                 )      0x00007FFE0EF87C10 - BaseThreadInitThunk
(unknown                                 )      0x00007FFE0FC6CE80 - RtlUserThreadStart

from pbrt-v4.

Interesting how you also get an "invalid PTX input" error, but instead of an argument mismatch you are getting a parsing error.

I will try moving to the 456.71 WHQL drivers instead of the 457.80 beta driver I am currently on, and also try directly loading the generated PTX file instead of using the generated embedded version, but that will be tomorrow or over the weekend.

from pbrt-v4.

(Sorry I haven't been able to be much help with this.)

I did notice you guys are using Optix 7.2; I've been on 7.1. I just tried 7.2 on Linux now and that worked fine, so it's not a 7.1 vs 7.2 issue. (I'm on nvcc 11.1 and driver 455.23.05 there, FWIW.)

One random thought is to try clearing the OptiX cache, which is in %LOCALAPPDATA%\NVIDIA\OptixCache apparently, just in case something's out of whack with what's stored there.

from pbrt-v4.

Good news (of a sort): I can reproduce this locally but it only happens in debug builds, which is interesting.

With 7.1, I get

[ 483317.000 20201022.170354 /home/mmp/pbrt-v4/src/pbrt/gpu/accel.cpp:556 ] ERROR OptiX: COMPILE FEEDBACK: COMPILE ERROR: Debug level OPTIX_COMPILE_DEBUG_LEVEL_FULL requires optimization level OPTIX_COMPILE_OPTIMIZATION_LEVEL_0.

which is weird, since looking at the code, that's definitely what it's setting the optimization level to.

With 7.2, I get the following, too.

[ 487875.000 20201022.170649 /home/mmp/pbrt-v4/src/pbrt/gpu/accel.cpp:556 ] ERROR OptiX: COMPILE FEEDBACK: COMPILE ERROR: Invalid PTX input: ptx2llvm-module-001: error: Failed to translate PTX input to LLVM
ptx2llvm-module-001, line 9; warning : Unsupported .version 7.1; current version is '6.4'
Call parameter type does not match function signature!
  %24 = load [8 x i8]* %param0, !dbg !1844
 [0 x i8]  %25 = call i32 %23([8 x i8] %24), !dbg !1844

I will dig into this first thing tomorrow morning. (I may start some bisecting now to try to narrow down when it broke.)

from pbrt-v4.

I was initially on OptiX 7.1.0 and CUDA 11.1, but since OptiX 7.1.0 does not list CUDA 11.1 as a supported toolkit, I switched to OptiX 7.2.0 but still ran in similar issues.

Great to hear you were able to reproduce it! I did not thought about trying a release build instead of a debug one.

from pbrt-v4.

I'm not sure if is related with this issue.. but before open a new issue I prefer comment here. ( can delete the entry if not related..)
I uses CMake 18.1 / OptiX 7.20 / CUDA 11.0 and MSVC 2019 with the last updates.
After a success 'Configurate' project in CMake UI, when try to 'Generate' the project, get a lot of warning messages for each 'sub-module' like this..

CMake Warning (dev) in CMakeLists.txt:
  Policy CMP0104 is not set: CMAKE_CUDA_ARCHITECTURES now detected for NVCC,
  empty CUDA_ARCHITECTURES not allowed.  Run "cmake --help-policy CMP0104"
  for policy details.  Use the cmake_policy command to set the policy and
  suppress this warning.

  CUDA_ARCHITECTURES is empty for target "soac".
This warning is for project developers.  Use -Wno-dev to suppress it.

..this include the direct related Cuda project:
CUDA_ARCHITECTURES is empty for target "optix.cu".
Even this messages, the project compiles Ok.. but fail in the run time.
Cheers..

from pbrt-v4.

It should not be a problem. Since 3.18, CMake will automatically pass along to the CUDA compiler the proper flags for compiling for specific CUDA GPU architectures, but PBRT uses the earlier way of specifying the flags manually. Since PBRT does not require a version that includes this new behaviour, that it does not specify which of the old or new behaviour should be assumed, and you are using a version that supports the new behaviour, you end up with that warning.
More information about the policy can be found on this page.

That warning could be silenced by telling CMake that the old behaviour is to be used, by adding cmake_policy(SET CMP0104 OLD) to the CMakeLists.txt file. I do not know what happens when using a CMake version which predates the mentioned policy: do you get an error or warning for the unknown policy, or does it just ignore it?

from pbrt-v4.

That warning could be silenced by telling CMake that the old behaviour is to be used, by adding cmake_policy(SET CMP0104 OLD) to the CMakeLists.txt file. I do not know what happens when using a CMake version which predates the mentioned policy: do you get an error or warning for the unknown policy, or does it just ignore it?

I only have CMake 3.18 installed, then don't know the result with other versions. Currently, I ignore this warning since the project compiles without errors. I shared it here because it could have relation with the runtime error...

from pbrt-v4.

do you get an error or warning for the unknown policy, or does it just ignore it?

This was not meant as a question to you; sorry, I should have formulated it as "does one get an error or warning […]" instead. But it is a question I am wondering about and due to not knowing the answer I have been reluctant to send a patch to add that cmake_policy(SET CMP0104 OLD).

It really should not have any impact on the code as pbrt manually provides all the necessary flags regarding GPU architectures the code should be built for, so the fact that CMake does nothing in that regard is fine. I have been having that warning for some time now, even when I could run PBRT just fine, but it is definitely annoying.

from pbrt-v4.

Interesting! I was trying to do something similar by commenting out that if-statement and always use the else part, but ran into some compilation with nvcc complaining that the host function _fdtest can not be called from device code; still looking into that.

Mmh, I will look into that build dependency issue. Could you please open an issue for it, to make sure I do not forget about it?

The bad news is that I haven't been able to figure out how to do that for just the PTX build, so the diff below fixes it, but at the cost of not having debugging symbols in any of the other stuff built with the CUDA compiler, which is undesirable...

Yeah… Looks like I really need to get going with those CMake improvements I have been talking about. I will see what I can get over the weekend.

from pbrt-v4.

@mmp The following change seems to work for me, regarding the build dependency issue; AFAICT the current code should be working fine, but there is something weird happening underneath that makes it not work. I'll open an MR shortly.

diff --git a/CMakeLists.txt b/CMakeLists.txt
index e7bd6c7..152e154 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -245,7 +245,7 @@ if (CMAKE_CUDA_COMPILER)
               "-DOUTPUT=${embedded_file}"
               -P ${CMAKE_CURRENT_SOURCE_DIR}/cmake/bin2c_wrapper.cmake
             VERBATIM
-            DEPENDS "${lib_name}"
+            DEPENDS "${lib_name}" $<TARGET_OBJECTS:${lib_name}>
             COMMENT "compiling (and embedding ptx from) ${cuda_file}"
           )
           set (${output_var} ${embedded_file})

from pbrt-v4.

Something like the following seem to do the job:

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 103dfa6..6071c78 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -112,6 +116,8 @@ endif ()
 #########################################
 ## CUDA / OptiX

+add_library (cuda_build_configuration INTERFACE)
+
 include (CheckLanguage)

 check_language(CUDA)
@@ -150,14 +156,23 @@ if (CMAKE_CUDA_COMPILER)
         set (PBRT_CUDA_DIAG_FLAGS "${PBRT_CUDA_DIAG_FLAGS} -Xcudafe --diag_suppress=declared_but_not_referenced")
         # WAR invalid warnings about this with "if constexpr"
         set (PBRT_CUDA_DIAG_FLAGS "${PBRT_CUDA_DIAG_FLAGS} -Xcudafe --diag_suppress=implicit_return_from_non_void_function")
-        set (PBRT_CUDA_DIAG_FLAGS "${PBRT_CUDA_DIAG_FLAGS} --expt-relaxed-constexpr")
-        set (PBRT_CUDA_DIAG_FLAGS "${PBRT_CUDA_DIAG_FLAGS} --extended-lambda")
         set (CMAKE_CUDA_FLAGS "${CMAKE_CUDA_FLAGS} ${PBRT_CUDA_DIAG_FLAGS}")

-        # Willie hears yeh..
-        set (CMAKE_CUDA_FLAGS "${CMAKE_CUDA_FLAGS} -Xnvlink -suppress-stack-size-warning")
-
-        set (CMAKE_CUDA_FLAGS "${CMAKE_CUDA_FLAGS} --forward-unknown-to-host-compiler")
+        target_compile_options (
+            cuda_build_configuration
+            INTERFACE
+                $<$<COMPILE_LANG_AND_ID:CUDA,NVIDIA>:
+                    --std=c++17
+                    --use_fast_math
+                    # The "$<NOT:$<BOOL:$<TARGET_PROPERTY:CUDA_PTX_COMPILATION>>>" part is to not add debugging symbols when generating PTX files for OptiX; see https://github.com/mmp/pbrt-v4/issues/69#issuecomment-715499748.
+                    $<IF:$<AND:$<CONFIG:Debug>,$<NOT:$<BOOL:$<TARGET_PROPERTY:CUDA_PTX_COMPILATION>>>>,-G;-g,-lineinfo;-maxrregcount;128>
+                    --expt-relaxed-constexpr
+                    --extended-lambda
+                    # Willie hears yeh..
+                    -Xnvlink;-suppress-stack-size-warning
+                    --forward-unknown-to-host-compiler
+                >
+        )

         # https://wagonhelm.github.io/articles/2018-03/detecting-cuda-capability-with-cmake
         # Get CUDA compute capability
@@ -172,13 +187,6 @@ if (CMAKE_CUDA_COMPILER)
                          RESULT_VARIABLE CUDA_RETURN_CODE
                          OUTPUT_VARIABLE CHECK_CUDA_OUTPUT)

-        set (CMAKE_CUDA_FLAGS "${CMAKE_CUDA_FLAGS} --std=c++17")
-        if (CMAKE_BUILD_TYPE MATCHES Debug)
-          set (CMAKE_CUDA_FLAGS "${CMAKE_CUDA_FLAGS} --use_fast_math -G -g")
-        else()
-          set (CMAKE_CUDA_FLAGS "${CMAKE_CUDA_FLAGS} --use_fast_math -lineinfo -maxrregcount 128")
-        endif ()
-
         if (NOT ${CUDA_RETURN_CODE} EQUAL 0)
             message (SEND_ERROR ${CHECK_CUDA_OUTPUT})
           else ()
@@ -233,6 +241,7 @@ if (CMAKE_CUDA_COMPILER)
           endif()
           target_include_directories ("${lib_name}" PRIVATE src ${CMAKE_BINARY_DIR})
           target_include_directories ("${lib_name}" SYSTEM PRIVATE ${NANOVDB_INCLUDE})
+          target_link_libraries ("${lib_name}" PRIVATE cuda_build_configuration)
           add_dependencies ("${lib_name}" pbrt_soa_generated)
           set (c_var_name ${output_var})
           set (embedded_file ${cuda_file}.ptx_embedded.c)
@@ -718,6 +723,10 @@ endif ()

 target_compile_options (pbrt_lib PUBLIC ${PBRT_CXX_FLAGS})

+if (PBRT_CUDA_ENABLED)
+    target_link_libraries (pbrt_lib PRIVATE cuda_build_configuration)
+endif ()
+
 add_sanitizers (pbrt_lib)

 if (WIN32)

Build is still ongoing, but I checked the content of the generated build.ninja and saw that optix.cu does not get the "-G -g" while other CUDA files do get them, in debug builds.

from pbrt-v4.

Let me apply and try again..

from pbrt-v4.

For which scene are you seeing that? (I'll see if I can reproduce it here.)

from pbrt-v4.

@mmp killeroo-gold.pbrt; I opened #72 to track it.

I opened #71 for the CMake diff I posted above, but left it as a draft to gather some feedback from others and make sure it somehow does not break something else.

from pbrt-v4.