[Bug Report] Bug in rendering camera images in Gym environment about orbit HOT 4 CLOSED

Yes, I want to run the camera in headless mode to collect data.
Here are some of my attempts and results

• I modified the train code as you said, but I still get the same error

# launch the simulator
config = {"headless": args_cli.headless}
# load cheaper kit config in headless
if args_cli.headless:
    app_experience = f"{os.environ['EXP_PATH']}/omni.isaac.sim.python.gym.headless.kit"
else:
    app_experience = f"{os.environ['EXP_PATH']}/omni.isaac.sim.python.kit"
# launch the simulator
simulation_app = SimulationApp(config, experience=app_experience)

# revision
from omni.isaac.core.utils.extensions import enable_extension
# enable viewport extension if not running in headless mode
enable_extension("omni.kit.viewport.bundle")

• I tried to stop loading "omni.isaac.sim.python.gym.headless.kit" and the good news is that the code works fine, but I checked the image data I got and I found that none of them changed, but in non-headless mode I can get several frames of changing image data. Is this because I didn't load "omni.isaac.sim.python.gym.headless.kit"?

Actually I just want to get the camera data in headless mode, because my environment has very low frame rate in non-headless mode, I hope to provide a demo of using the camera to collect data in headless mode, thanks a lot.

from orbit.

Are you running this in headless mode?

If so, the above changes will try loading the configuration file for Isaac Sim called "omni.isaac.sim.python.gym.headless.kit". This configuration file does not have the viewport extension enabled by default, which is needed for rendering camera images. We enable it explicitly in the RL environment if running in non-headless mode:

https://github.com/NVIDIA-Omniverse/Orbit/blob/main/source/extensions/omni.isaac.orbit_envs/omni/isaac/orbit_envs/isaac_env.py#L413

You can add these lines right after launching the simulator. They'll enable the extension manually.

from omni.isaac.core.utils.extensions import enable_extension

# enable viewport extension if not running in headless mode
enable_extension("omni.kit.viewport.bundle")

I think it would make sense to enable this explicitly in the camera class to avoid unnecessary duplication of this code at multiple places.

Let me know if the issue gets fixed with it.

from orbit.

Is it just with play_camera.py? I have a unit test that runs in headless mode and after a few initial blank frames (a known issue in the simulator), it renders fine. You can run the test script here for it:

./orbit.sh -p source/extensions/omni.isaac.orbit/test/test_camera.py

from orbit.

Thanks, I ran test_camera.py and the camera still collects data normally in headless mode, but not in RL frame. I found the reason, it's because sim.step(render=False) is not collecting camera data, just set render=True to solve the problem. Thank you for your answer.

from orbit.