any recommended prebuilt interior setups? (default setup is outdoors, but also need to mimic indoor lighting scenarios) about com.unity.perception HOT 4 CLOSED

Hi @lessw2020!

Sorry for the later response than usual. Here's another info dump that may help with understanding the available options here. In short, creating a randomized interior scene is possible, but choosing what scene properties to randomize and configuring interior lighting can be tricky. And here's why...

Interior randomization

Choosing the scope of randomizations you want to simulate here is key. On one hand, you can just randomize the colors of a 4 walled empty room, place your object in the center and take a picture. On another hand, you could randomize the wall textures, room layout, furniture and clutter objects, and light sources and end up with a pretty hefty project to implement, not to mention the extensive asset acquisition effort needed to source the objects to fill up your rooms. Based on your question, if you're only interested in basic lighting configurations, maybe the 4 walled empty box is the best place to start.

Interior lighting

A project's lighting configuration probably has the greatest influence over the final rendered output over any other simulation property. Unity has many lighting options each of which is designed as a different trade off between performance and realism/capability. I will list the 3 most pertinent options that you will likely be interested in.

1. URP baked lighting - The universal render pipeline offers the most performant lighting configurations by using an offline baking process to generate realistic bounce lighting within a static scene and then using simple shadow mapped dynamic lights + light probes to make dynamic (or randomized) objects "fit" into the baked scene. This option provides high FPS, but lacks the visual fidelity needed for interior environments and is geared toward more outdoor like settings. Also, depending on scene randomization complexity, light baking might not be the best option. Randomly generated scenes will often place objects and adjust lighting in ways that make the new scene incompatible with the original baked lighting configuration.

2. HDRP lighting - A default HDRP scene offers a step toward more realistic environments with a much larger array of lighting settings (soft shadows, multiple dynamic lights, etc.) and a host of additional real time effects like camera exposure and screen space ambient occlusion. A warning though: real time screen space effects may make your scene "look better", but the way these effects are calculated is not based on how light works in the real world, so realism may vary. Another huge advantage of HDRP is the potential to have moderately realistic lighting without baking your lighting configuration (though you can integrate light baking if you want to). However, there is no real time global illumination option in default HDRP, meaning that your scene will not simulate complex real world light behavior such as light bouncing, light bleeding, or realistic shadows for dynamic scenes. This can result in unrealistically dark scenes when only using directional lights and windows (without extra interior lights to brighten things up). Overall though, HDRP offers a good compromise between performance and realism for some use cases.

3. HDRP DXR (DirectX Raytracing) - Unity offers some preview ray tracing features in its latest editor versions that can be used to drastically improve the realism of your scene... for a cost. Here are the pros and cons of DXR:

Pros:

• Can simulate more realistic light behaviors (light bouncing, light color bleeding, and realistic shadows)
• No light baking required

Cons:

• Requires special hardware to run (Nvidia RTX graphics cards required)
• Time consuming to render (relative to default HDRP). Some lighting options (Global Illumination) are less expensive then others (Path tracing).
• More complicated to configure
• These features are still in preview and subject to change

Here's a visual comparison of the different lighting configurations:

1. Default HDRP with two lights (one directional light and one dim ceiling point light)
   
2. HDRP + Global Illumination (GI) (notice how much brighter the scene is with ray traced light bouncing)
   
3. HDRP + Path Tracing (128 samples) (notice the red light bleeding from the cube onto the floor and the increased shadow quality)
   
4. HDRP + Path Tracing (4096 samples) (more samples = less ray tracing noise = longer time to render)
   

Feel free to ping me if you need any additional clarification on the above information. Also, you may want to consider contacting us directly at [email protected] to setup a time to chat more extensively on this topic and discuss what other tooling we may have available to help accelerate development here.

from com.unity.perception.

Hi @sleal-unity!
Thanks a bunch for all the info above and for generating the comparison images. For our use case, we'd definitely like to go with max realism. The path tracing mimics a lot the effects we see in our real shadowed images, so nice to see the image.
I am happy to note that the HDRP shadows, have already been very helpful to train the classifiers to ignore real shadow, so synthetic is already helping. Thus, making it as photorealistic as possible is the ultimate goal.
Also I did the basic put up four walls to make a basic room - longer term the idea is to make a series of 'scenario' rooms to generate robust simulations in various settings and thus produce well rounded training sets.
I'll ping you via the direct line as looking at purchasing new hardware so what to make sure it can use the path tracing (ie. ampere gpu architecture).
Thanks again!

from com.unity.perception.

Sounds good, thanks again @sleal-unity

from com.unity.perception.

@lessw2020, one really important thing I need to confirm is whether the perception package's semantic labeling shaders and such are compatible with HDRP's preview features (GI, path tracing, etc.). But regardless, let's chat more through the [email protected] email list about what would be involved in taking these next steps toward more realistic environments.

from com.unity.perception.