Warnings are temporarily disabled. We should re-enable them and fix the issues.

We should be able to quantify

• Depth precision
• Vertex transform precision

Omniverse does 64-bit matrix math under the hood, so theoretically could support techniques like Relative To Center for high precision rendering. My suspicion is that the RTX pipeline (including BVH construction, ray tracing) is different enough that it may not have this, but we should check.

If not, we may need to implement origin rebasing similar to Cesium for Unreal, or sub-scenes in Cesium for Unity.

Background reading:

• Relative To Center (RTC): https://help.agi.com/AGIComponents/html/BlogPrecisionsPrecisions.htm
• Cesium for Unity: https://cesium.com/learn/unity/unity-placing-objects/
• Cesium for Unreal (pre UE5): CesiumGS/cesium-unreal#80

Several Nvidia libraries are needed to build Cesium Omniverse - Nvidia's USD fork, Python 3.7, and likely Omniverse Client once we start interacting with Nucleus. Currently we require users to install Connect Sample 200.0.0 and run build.bat which will fetch these libraries from Pacman. Then our CMake project links to these libraries.

From the README:

Install Connect Sample 200.0.0 (use this exact version) with the default install location. The reason we need Connect Sample is because it downloads some dependencies that we need including USD and OmniClient. In the future we should download these dependencies ourself.

This approach was inspired by https://github.com/PatrickPalmer/Omniverse-Connect-cmake which was inspired by https://forums.developer.nvidia.com/t/creating-an-omniverse-usd-app-from-the-connect-sample/189557.

Super easy way to get started, but we should be fetching the libraries as part of our CMake build process. We should poke around the Connect Sample 200.0.0 code to see how it's done there.

If a glTF uses the KHR_materials_unlit extension we should render with an unlit material and not generate normals.

We could also give some control to the user. There could be a drop-down list to override the default behavior. The options would be UNLIT, LIT, or AUTO.

Currently we are using a branch of cesium-native with some hacks to remove terrain skirts and skip texture decoding.

https://github.com/CesiumGS/cesium-native/tree/nowarn

Eventually we should either bring these changes back into cesium-native main or fix the problems on the Cesium for Omniverse side.

When building a static debug library with coverage enabled on Linux we see the following error in Omniverse Code:

2022-11-02 19:06:52  [Error] [omni.ext.impl.custom_importer] Failed to import python module cesium.omniverse. Error: ...snip.../cesium/omniverse/bindings/cesium/omniverse/CesiumOmniversePythonBindings.cpython-37m-x86_64-linux-gnu.so: undefined symbol: llvm_gcda_emit_arcs. Traceback:

To reproduce:

cmake -B build-debug -D CMAKE_C_COMPILER=gcc-9 -D CMAKE_CXX_COMPILER=g++-9 -D CMAKE_BUILD_TYPE=Debug -D CESIUM_OMNI_ENABLE_COVERAGE=ON -D BUILD_SHARED_LIBS=OFF
cmake --build build-debug --parallel 8
cmake --install build-debug --prefix ../cesium-kit-exts/exts/cesium.omniverse/cesium/omniverse/bindings/cesium/omniverse/ --component kit
# Then load the extension in Omniverse Code

This seems like a problem with pybind11. One way to work around it is to set -D BUILD_SHARED_LIBS=ON. Unfortunately this triggers a different bug in the build system: #30.

USD currently does not support depth-bias for gPrims, so all meshes are rendered from back to front. This is an issue for us when we start working with overlaying construction sites and other props that may have sections that go lower than the tileset.

Autodesk opened an issue and pull request for a similar need in 2021 but the PR was abandoned as the Pixar team had some issues with it. The USD team is definitely open to the idea but they need someone to champion it.

We should be that champion, as this will continue to be a major issue for us.

According to https://forums.developer.nvidia.com/t/usd-coordinate-system/201914:

• USD stage is right-handed
• USD stage coordinate system can be Y-up or Z-up (default is Y-up)
• USD stage meters-per-unit is configurable (default is 0.01, a.k.a centimeter units)

3D Tiles is right-handed, Z-up, and meters (from the spec)

Special care should be taken so that 3D Tiles are positioned/scaled correctly in the scene

Centimeters are currently hard coded and we need to make this more generic. A good solution would be to check the USD state for the unit and change our tiles respectively.

The default far plane distance is too small when we create a scene with centimeter units. Our kit extension should be able to adjust this setting dynamically.

When we are using OmniPBR, there is a brief flash of white while tiles are loading. This does not appear to be related to the synchronous material loads problem we were having earlier.

This project is composed of a few different parts:

• C++ library for streaming 3D Tiles into a USD stage
• Python library that wraps the C++ streaming library
• Omniverse Kit extension - front-end UI that can be added to Omniverse Code, Omniverse Create
• ... future libraries
• ... future kit extensions

Are these versioned separately or together? The kit extension version is here and the change log is here. This is what's shown when loading the extension in Omniverse Code/Create. Meanwhile the C++ library version is here. If we want to version everything together we need to be careful that these don't get out of sync.

I think it makes more sense if the C++/Python library are versioned separately from the Kit extensions since the C++ library may have users outside of the Omniverse ecosystem.

Most of our testing has been with assets hosted on ion. We should make sure that local tilestes work too. This includes:

• Local servers such as http-server
• file URI schemes like file:///

Debugging is currently hacky. The solution found for #1 works but we could do better when we have time.

More details TBD.

There are some problems with UsdPreviewSurface in Omniverse:

• Transform2d node doesn't seem to work (we need this for raster imagery)
• Strange behavior when using primvars for UV coordinates
• opacity below 1.0 causes model to become completely transparent

Instead of going much further in this direction we should use MDL directly. OmniPBR template is a good starting point. It has texture translate/scale controls that we need to finish implementing raster overlays #15.

Resources:

• https://docs.omniverse.nvidia.com/prod_materials-and-rendering/prod_materials-and-rendering/materials.html
• https://forums.developer.nvidia.com/t/need-help-with-textured-mdl-materials-through-omniverse-connector/202769

Here's the branch where I tried to get Transform2d working: https://github.com/CesiumGS/cesium-omniverse/commits/material-improvements

We should set up Github Actions in this repo. Each job would:

• Build Conan libraries (and cache them)
• Fetch Nvidia libraries #7 (and cache them)
• Configure and build
• Check clang-format
• Check clang-tidy
• Run tests
• Run coverage
• Generate documentation
• Package

We would set this up for Ubuntu GCC and Windows MSVC. For compatibility with older GLIBC systems we might need to have a CentOS Docker container do the packaged release builds. See how this is done in our internal repos.

Other thoughts:

• Not sure what to do for Omnvierse Kit extensions

Related to #3 and #4

We should have a lightweight Python API that wraps the C++ 3D Tiles streaming API.

This is the API used by our Omniverse Kit extensions. It should also be generic enough that it can be used by other Kit extensions and even non-Omniverse Python code.

Currently we use pybind11 to generate a Python Dynamic Module (.pyd), which includes some baked-in documentation. See PythonBindings.cpp. In addition to this we probably need some basic .py wrapper code to create a proper Python module.

We should consider providing an ABI stable C interface for CesiumOmniverse, inspired by the concept of Omniverse Native Interfaces (ONI) as described in the Carbonite SDK documentation.

The initial skeleton of this can be found in CesiumOmniverse.h. We would need to think of an elegant way to pass through USD objects such as UsdStageRefPtr and GfMatrix4d.

This also affects the Python API which is essentially just a wrapper of the C++ API. If we switched to a C API we could consider switching from pybind11 to cython.

Smooth normals look bad on tile edges where there are terrain skirts.

Ways to fix this:

• If we have control over normal generation, generate hard normals for terrain skirts and smooth normals for the interior triangles
• Disable terrain skirts in cesium-native (that's what nowarn was doing)
• Unlit materials: #36

• Decide how to package the C++ and Python libraries. As far as tooling goes CPack has worked well for at least one internal Cesium project. This should be straightforward since we already have an install target that installs the libs and headers. Packaged builds would be hosted on the GitHub releases page.
• Should the C++ std library be statically linked or not? I can't find the link anymore, but I believe this was recommended somewhere in the Carbonite documentation. Currently it's dynamically linked which means users would need to install the redistributables.
• Decide how to pack the Omniverse Kit extensions. Note that the extension bundles the .pyd file and its dependencies, which includes the USD libraries and Python 3.7. Is this considered ok practice? How do we get these onto Omniverse Exchange?

The Omiverse Kit preview image should be changed before we release the extension. Currently it's a debug screenshot.

If the plugin is set to autoload and you hit Create Tileset in the menu in CREATE, we Crash hard due to externals.pAssetAccessor being null in CesiumIonTilesetLoader.cpp.

Omniverse Code will automatically hot-reload extensions when files that match the fswatcher pattern are changed. This works pretty well for Python code. Hot reloading C++ code is more complicated for several reasons:

• We may need to clear the USD stage after each hot-reload, or some other "reset" behavior
• The C++ code is built as a Python Dynamic Module (.pyd) — essentially a DLL — and the Python interpreter within Omniverse Code will hold onto this until program exit. Therefore trying to replace the file at runtime with a new version is not permitted by Windows.

One possibility is to handle the hot-reloading completely within the DLL's. On every update CesiumOmniversePythonBindings.cp37-win_amd64.pyd could check if a new variant of CesiumOmniverse[N].dll has been created and dynamically load it.

Also see:

• https://discuss.python.org/t/how-to-unload-native-extensions-pyd-on-windows/3749/2
• https://stackoverflow.com/questions/59906132/reloading-cython-modules-on-the-fly

• Docs: https://mypy.readthedocs.io/en/stable/
• VS Code extension: https://marketplace.visualstudio.com/items?itemName=matangover.mypy
• Visual Studio support: https://learn.microsoft.com/en-us/visualstudio/python/editing-python-code-in-visual-studio?view=vs-2022

The georeference location should not be hardcoded in the C++ code (see here). Instead it should be configurable in the UI or automatically computed based on the tileset bounding volume.

We should hook into the Omniverse logging system instead of using spdlog.

See http://omniverse-docs.s3-website-us-east-1.amazonaws.com/carbonite/137.0-pre/docs/omni.log/Logging.html

Like https://github.com/CesiumGS/cesium-unreal-samples but for Cesium for Omniverse.

Each sample project would be a .kit application that uses the Cesium for Omniverse extension.

We can't run the extension on Linux right now due an HTTP status code error. We should fix this as soon as possible.

2022-11-02 19:45:31  [Info] [omni.usd] - Received status code 0 for asset response https://api.cesium.com/v1/assets/1/endpoint?access_token=...

tileset->getOptions().forbidHoles = true doesn't seem to work. Holes appear on the edges of the screen as the camera is rotating.

A similar thing happens in Unreal in editor mode but not play mode. According to @nithinp7

In Unreal, the holes on the edge of the screen come from a one-frame delay between the editor camera position as the renderer knows it vs the editor camera position as the tileset traversal is aware of it

Could something similar be happening in Omniverse?

Once we fix this, we should delete tileset->getOptions().enableFrustumCulling = false

Investigate whether it's possible to debug C++ library code while running Omniverse Code.

Some starting points:

• https://stackoverflow.com/questions/60440765/how-can-i-attach-to-a-specific-process-in-visual-studio-code
• https://stackoverflow.com/questions/46169522/how-to-debug-underlying-c-library-from-python-interface

main-old is the original demo branch of Cesium for Omniverse and we'd like to stop developing on it as soon as possible.

There are a few things we have to bring over to main before we can delete the branch:

• The USD asset resolver plugin. This allows us to load in-memory textures. To summarize the old readme we need to:
  • Add instructions for setting the PXR_PLUGINPATH_NAME environment variable and creating the mem.cesium file.
  • Install pluginInfo.json into the bindings directory. See CMakeLists.txt#L69-L73.
• Meters to centimeters conversion from c61ef95

Similar to other Cesium projects, we should maintain a ThirdParty.json file in this repo.

• The build system automatically generates ThirdParty.json but it does not include Nvidia libraries currently

While a prim's textures are loading it has bright blue placeholder material. This is really distracting when loading 3D Tiles.

There are few possible fixes here:

• Force USD to not mark a prim as loaded until textures are loaded
• Decode PNG/JPEG as raw RGBA and wrap in a DDS or other uncompressed format
• Save texture to disk prior instead of storing in memory. See if this changes USD behavior / performance.

If you try running usdrt-test on Linux you'll see the error:

2022-12-16 19:35:33 [49,101ms] [Error] [carb.scripting-python.plugin] RuntimeError: Failed to acquire interface: cesium::omniverse::ICesiumOmniverseInterface (pluginName: nullptr)

At:
  /home/slilley/Code/cesium-omniverse/exts/cesium.omniverse/cesium/omniverse/extension.py(51): on_startup
  /home/slilley/.local/share/ov/pkg/deps/82b461274b877b7d5e1a405066cfd47a/kernel/py/omni/ext/_impl/_internal.py(148): _startup_ext
  /home/slilley/.local/share/ov/pkg/deps/82b461274b877b7d5e1a405066cfd47a/kernel/py/carb/profiler/__init__.py(81): wrapper
  /home/slilley/.local/share/ov/pkg/deps/82b461274b877b7d5e1a405066cfd47a/kernel/py/omni/ext/_impl/_internal.py(197): startup
  /home/slilley/.local/share/ov/pkg/deps/82b461274b877b7d5e1a405066cfd47a/kernel/py/omni/ext/_impl/_internal.py(280): startup_extension
  PythonExtension.cpp::startup()(2): <module>

2022-12-16 19:35:33 [49,101ms] [Error] [omni.ext.plugin] [ext: cesium.omniverse-0.0.0] Failed to startup python extension.

This can be reproduced on main as well by just linking usdrt in src/core/CMakeLists.txt

Some observations so far:

• The libusdrt.scenegraph.plugin.so file that ships with kit is significantly smaller than the libusdrt.scenegraph.plugin.so that ships with packman. This is not true for Windows.
• usdrt is an Omniverse Native Interface but out plugin is Carbonite Interface. Normally Carbonite interfaces can declare dependencies on other Carbonite Interfaces but I'm not sure if Carbonite can declare dependencies on Omniverse Native Interfaces.
• There's a note in the Carbonite Interface doc that could be relevant.

    /// Note that because of some dynamic linking behavior on Linux, it is possible that on
    /// unload this module could be shutdown but not actually unloaded from memory.  This would
    /// prevent any global variables from being reset or cleaned up during C++ static
    /// deinitialization.  However, if the plugin were loaded into the process again later,
    /// C++ static initialization would not occur since the module was not previously unloaded.
    /// Thus it is important to perform any and all global and static cleanup explicitly here.

Cesium for Unreal and Cesium for Unity also support multiple raster overlays. We need this for GTM as well.

A bug in Macros.cmake is causing us to have to copy over the .so and .pyd files on linux.

https://github.com/CesiumGS/cesium-omniverse/blob/main/cesium-omniverse/cmake/Macros.cmake#L109

The C++ unit test infrastructure is set up but there aren't any meaningful unit tests yet. We're using doctest.

Since the C++ library interacts with a USD stage and has no hard dependency on Omniverse or even a 3D context, we can mock a lot of streaming behavior in a headless way.

We may need to write additional unit tests for the Omniverse Kit extensions (Python code) but it's not clear what those would be yet or what the tooling would look like.

When building a static debug library on Linux we see the following error in Omniverse Code:

2022-11-02 19:57:34  [Error] [omni.ext.python] ImportError: /home/slilley/Code/cesium-omniverse/cesium-kit-exts/exts/cesium.omniverse/cesium/omniverse/bindings/cesium/omniverse/CesiumOmniversePythonBindings.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN14CesiumGeometry3RayC1ERKN3glm3vecILm3EdLNS1_9qualifierE0EEES6_

pybind11 might be the culprit because if we build cesium-omniverse as a shared library the problem goes away. Likely related to #32. Another related issues is that shared libraries aren't installed correctly, CC #30.

The Omniverse Kit extension UI should resemble the Cesium for Unreal, Cesium for Unity, and Cesium for O3DE UIs so that there's a consistent user experience across engines.

• Ability to load tileset from path or asset id / access token
• Ability to change georeference origin
• Debugging aids
• ...

In order to get satellite imagery showing up on the globe we need to add raster overlay support.

To start, see how it's done in Cesium for Unreal: https://github.com/CesiumGS/cesium-unreal/blob/a68e0c25fc207b250f6a8be480bc77216b6fd2a2/Source/CesiumRuntime/Private/CesiumRasterOverlay.cpp#L98

We use cesium-native for 3D Tiles streaming, loading, and caching. Tiles are cached as glTF blobs in a local sqlite database.

Since we need to convert glTF to USD to bring it into Omniverse, it would be more efficient if we could store USD in the cache so that we don't have to perform this conversion every time a tile is loaded from the cache.

Related to and will likely build on the work in CesiumGS/cesium-native#566

The built-in sky environment works nicely at ground level but isn't accurate at planetary scale.

Ground level

Planetary scale

We use the Pylance extension in VS Code which provides code completion, code navigation, etc as we develop Omniverse Kit extensions.

In order to locate Omniverse python modules we include them in python.analysis.extraPaths. The problem is Pylance does not support globs so we have to list all the modules manually which is error prone:

This list was auto-generated when we first created the project from within Omniverse Code. The list for Linux was much smaller for some reason, which means we get pretty poor IntelliSense.

Either we need to wait for microsoft/pylance-release#2712 to be fixed or we could write a script to update these workspace files by walking the Omniverse python tree.

Using Ninja on Windows fails to build due to issues with paths.

[199/441] Building CXX object extern\draco\CMakeFiles\draco_compression_attributes_enc.dir\src\draco\compression\attributes\sequential_integer_attribute_encoder.cc.obj
  FAILED: extern/draco/CMakeFiles/draco_compression_attributes_enc.dir/src/draco/compression/attributes/sequential_integer_attribute_encoder.cc.obj 
  C:\PROGRA~1\MICROS~2\2022\PROFES~1\VC\Tools\MSVC\1433~1.316\bin\Hostx64\x64\cl.exe  /nologo /TP -DDRACO_CMAKE=1 -DDRACO_FLAGS_SRCDIR=\"C:/Users/AMorris/Projects/cesium-omniverse/cesium-omniverse/extern/cesium-native/extern/draco\" -DDRACO_FLAGS_TMPDIR=\"/tmp\" -DNOMINMAX=1 -D_CRT_SECURE_NO_DEPRECATE=1 -IC:\Users\AMorris\Projects\cesium-omniverse\cesium-omniverse\extern\cesium-native\extern\draco -IC:\Users\AMorris\Projects\cesium-omniverse\cesium-omniverse\extern\cesium-native\extern\draco\src -IC:\Users\AMorris\Projects\cesium-omniverse\cesium-omniverse\build-debug\extern\cesium-native\src\cesium-native-external-build /EHsc /Zi /Ob0 /Od /RTC1 -MDd /w44018 /w44146 /w44244 /w44267 /w44804 /showIncludes /Foextern\draco\CMakeFiles\draco_compression_attributes_enc.dir\src\draco\compression\attributes\sequential_integer_attribute_encoder.cc.obj /Fdextern\draco\CMakeFiles\draco_compression_attributes_enc.dir\ /FS -c C:\Users\AMorris\Projects\cesium-omniverse\cesium-omniverse\extern\cesium-native\extern\draco\src\draco\compression\attributes\sequential_integer_attribute_encoder.cc
C:\Users\AMorris\Projects\cesium-omniverse\cesium-omniverse\extern\cesium-native\extern\draco\src\draco\compression\attributes\sequential_integer_attribute_encoder.cc : fatal error C1083: Cannot open compiler generated file: '': Invalid argument

The C1083 error is often related to long file paths in this particular type of scenario and that seems to be the case here. However, enabling NTFS long paths does not fix this issue. Further investigation is required to determine if in fact long paths are the issue, and if so, how to remedy the situation.

Currently we're using Doxygen for C++ documentation.

• All public API functions should be documented
• How can we minimize duplication between C++ documentation and pybind11 documentation?

Show a credit display on the bottom of the screen similar to CesiumJS and Cesium for Unreal. We should support two modes for this:

Mode 1: Show credits in popup when "Data attribution" is clicked

Mode 2: Show credits on screen

Note - credits are HTML and can have images (see the Bing Maps logo). We'll need to investigate the best way to do this with the Omniverse Kit UI.

Note 2 - Omniverse supports viewport overlays. That should be helpful here.

We should use the USDRT API once it is available in Kit. This will allow us to bypass USD and have a faster path to the GPU. Our biggest performance bottleneck right now is converting tiles to USD.

Occasionally raster overlays will load incorrectly and be mismatched with their tiles and expected level. This appears to only happen when moving through the scene quickly, and it is inconsistent at best.

When meshes don't have normals they are generated automatically, either CPU or GPU depending on the render settings. GPU is the default but it has some pretty serious artifacts for Cesium World Terrain. Can we chose CPU programmatically?

CPU	GPU

This is the UI panel for selecting the method: