[Vulkan] Renderdoc layer changing image memory requirements affects programs relying on external memory about renderdoc HOT 8 OPEN

You should never enable ENABLE_VULKAN_RENDERDOC_CAPTURE manually. This is an internal implementation detail that should not be set by users - it is not documented anywhere and is not a supported way of using RenderDoc.

Does this problem reproduce if you launch your program from the RenderDoc UI as intended?

Also RenderDoc does not have any support for openxr so if this program or library requires extra API support for openxr that may be the cause of the problem. I'm not familiar with openxr at all to be able to say whether or not it requires anything specific.

With that said I'm not really clear on exactly what is going wrong. You listed a validation error, but it seems to be related to not allocating the right amount of memory for a dedicated allocation. Without context I'm not clear - is your application providing the wrong size? Or is that an allocation coming from inside RenderDoc? You also mention an error message that comes from your library monado but I'm not sure how to interpret the message as there is only one size returned from vkGetImageMemoryRequirements.

If your application is not providing the right size of allocation then you will need to fix that - you should make sure you query for the image memory requirements and allocate the appropriate memory for it.

I don't believe I can follow your steps to reproduce if it requires openxr hardware, though I am not familiar at all with openxr or this monado library so if it is usable without any special hardware I can try to reproduce it.

from renderdoc.

Monado doesn't require OpenXR hardware, it defaults to a simple simulator with no hardware.

To go to a higher abstraction level, when an OpenXR application connects to monado-service (different processes), monado-service allocates VkImages, exports them with VK_KHR_External_memory_fd sends the fd over to the OpenXR application process where the fd is imported, and the OpenXR application renders to it. Every multi process application that exports and imports with VK_KHR_External_memory_fd is probably affected.

from renderdoc.

Does this problem reproduce if you launch your program from the RenderDoc UI as intended?

Same output messages in the terminal where qrenderdoc was executed when running the program through there.

Also RenderDoc does not have any support for openxr so if this program or library requires extra API support for openxr that may be the cause of the problem.

I don't think there's anything special about OpenXR as far as the Vulkan usage is concerned, it's just a regular application using IPC mechanisms from published Vulkan extensions.

You also mention an error message that comes from your library monado but I'm not sure how to interpret the message as there is only one size returned from vkGetImageMemoryRequirements

There are two programs running Vulkan and sharing images using a file descriptor. The allocation occurs when one application tries to import an image created by the first. The first size is the size returned by vkGetImageMemoryRequirements in the application importing the image, the second is the size from when the image was created on the server-side (also calling vkGetImageMemoryRequirements, but on the server). In this case I'm only capturing the client process, hence the mismatch.

As far as I can tell, the error in the validation for dedicated allocation happens because Renderdoc is modifying the value returned by vkGetImageMemoryRequirements, so even though the allocation size is the one returned by vkGetImageMemoryRequirements, as seen by the application (see the code that allocates the image), it is not the size that the validation layer expected.

I don't believe I can follow your steps to reproduce if it requires openxr hardware, though I am not familiar at all with openxr or this monado library so if it is usable without any special hardware I can try to reproduce it.

Sorry about that, I would have shared a pure Vulkan program using those features if I had one at hand 😅

Is it possible to easily check the VkDriverInfo that Renderdoc collects? Skimming through the code, it doesn't look like Renderdoc is intended to change the return value of vkGetImageMemoryRequirements except on "old" official AMD drivers.

from renderdoc.

OK I can try to reproduce sometime soon then. Without any other knowledge I was going by what the original reporter said that I would need some hardware that monado supports.

I understand now what you're doing at a high level but I'm still not exactly clear what the problem is. Is the only issue that the memory requirements being reported are different when running under RenderDoc than not? That is not unusual but it should not cause any problems for a valid application - you can't expect a specific size from the vulkan API. There are guarantees about alignment etc as well as invariance, so you could require a minimum but not an exact value or a maximum, whether created normally or imported from one of the external memory extensions.

If your program is coded around expecting a particular memory size then that sounds like a problem. The vulkan spec does not guarantee that and applications are expected to query the driver for how much memory is required before binding it.

from renderdoc.

It's entirely possible that monado's checks are stricter than necessary.

Tracing the code a bit there is one code path where this happens and that is when importing an image that has been elsewhere created (i.e. in monado-service). https://gitlab.freedesktop.org/monado/monado/-/blob/8a4963f719ad7f1c8622e9361df52da483c68317/src/xrt/auxiliary/vk/vk_helpers.c#L818-822

Presumably max_size is the size of the originally allocated memory (in monado-service, not running under renderdoc), and the memory requirements that are checked here are queried in the openxr client side running under renderdoc. Renderdoc pads the memory requirements and the check fails.

I'm not sure what the check was originally intended to do but I don't see max_size actually used, so is this a pointless check we should remove?

from renderdoc.

RenderDoc doesn't pad memory requirements, but yes an application can't make assumptions about the size required for any resource beyond the limited guarantees the spec provides. That's why I was asking above about what exactly the problem is since it's not clearly explained what (if anything) is going wrong in terms of legal vulkan API behaviour vs. internal error messages.

from renderdoc.

I was able to set up an environment and build the dependencies to reproduce this. I ran into that internal check mentioned above, which looks to me to be invalid and I removed that.

After that I didn't encounter any problems or crashes. I did see the validation error and found the cause of that, though the application should not be able to see that error so it shouldn't cause any problems either. I've put in a workaround to prevent it from firing but generally speaking I don't recommend using validation with RenderDoc as it's subject to potential false positives and false negatives. You should run validation with your application on its own separately from RenderDoc to be sure you get accurate results.

So I've still not found any actual bug short of the validation error which as above should not be application-visible to cause any more problems than a false positive if you capture API validation messages. The validation error is technically correct but was causing no practical harm I believe, I've opened an issue within khronos to see if the requirement can be loosened and then the workaround can be removed.

I'll leave this open for now but please clearly explain what the actual bug is (if there is any bug aside from the extra validation error) and how to reproduce it.

from renderdoc.