glue-viz / glue-ar Goto Github PK

Python 98.75% JavaScript 0.11% CSS 0.86% HTML 0.28%

glue-ar's Introduction

glue-ar

This package is an experimental plugin for glue that allows exporting augmented reality (AR) figures out of the 3D scatter and volume viewers from glue-vispy-viewers. Currently this ability is exposed via viewer tools that save a 3D file representing the current view. Currently supported file formats include glTF/glB and USD.

Installation

This package is not (yet) listed on PyPI, so to install you'll need to clone this repository and install from source.

git clone https://github.com/Carifio24/glue-ar
cd glue-ar
pip install .  # Use `pip install -e .` to install in editable mode

Installation requires Node.js to be installed on your system, as we currently use JavaScript packages for performing Draco and Meshopt compression. (Having Node installed is all that you need - the npm/JS management relevant for the package is all handled by the package build process).

Viewer tools

The AR export tools are exposed in the toolbar as subtools of the "save" meta-tool, and can be accessed from its dropdown menu.

Sharing figures

model-viewer

If glTF is selected as the output file format, an HTML file is exported in addition to the glTF. This HTML file provides a display of the exported figure using the model-viewer web component. This model-viewer page provides a THREE.js-powered glTF viewer that, on an Android or iOS device, allows viewing the glTF file using the hardware AR capability. Additionally, this exported HTML has no additional dependencies and can be served using static file hosting.

glTF compression

The files exported by glue-ar can, in their original form, be quite large. In order to mitigate this problem for glTF files, glue-ar allows using Draco compression via the gltf-pipeline package, or Meshopt via the gltfpack package. These compression methods allow for a considerable reduction in file size (often an order of magnitude or more), and files compression by both methods can be read by model-viewer.

CoSpaces

Another popular option for sharing 3D files in augmented reality is CoSpaces, which allows viewing 3D files in a browser or on a mobile device via a dedicated app. The CoSpaces app allows viewing figures in AR on a flat surface directly, or using the Merge Cube to allow for a more tangible AR experience.

CoSpaces supports the glTF file format, so the outputs of glue-ar can be used in CoSpaces without modification. It is our aim to eventually allow automatic CoSpaces upload, but for now sharing your AR figures to CoSpaces requires some manual steps (as well as a CoSpaces account).

To create a scene with your newly-exported figure, do the following:

Go to the CoSpaces website and log in to your account
On the left side menu, navigate to "CoSpaces"
Click the "Create CoSpace" button
Select the environment you want:
- 3D environment > Empty scene for tabletop AR
- MERGE Cube > Empty scene to use the MERGE Cube (note that creating a MERGE Cube requires a plan addon)
In the bottom left corner, select Upload > 3D models, then press the Upload button to the right
Select your 3D model
- Note that CoSpaces currently does not support the extensions needed for Draco or Meshopt compression
Done!

glue-ar's People

Contributors

Watchers

Forkers

carifio24

glue-ar's Issues

USD scatter export doesn't support error bars

Currently the USD scatter export doesn't support error bars. For the GL exports we use the LINES rendering mode (which unsurprisingly matches the vispy lines quite nicely). AFAICT there isn't a corresponding primitive type for USD. Currently we just ignore error bars in this case, but we should come up with some way to support these. We have functionality for defining cylinder meshes, so one possible path is to just use very thin (defined in some appropriate way) cylinders to create the error bars.

Determine more robust method for sphere size

The sphere sizing in the scatter layer export defined here should get another look. This is just something that seemed to work well for the initial examples that I was looking at, and I'm not sure how specific it is to e.g. the bounds of the data I was looking at.

Add testing

It would be good to add some tests - unit testing for the utility functions and perhaps some tests of example output files.

Explore removing PyVista dependency

While PyVista has generally worked well for us so far for exporting, there are some issues with how the isosurface glTF files get autoconverted to USDZ by model-viewer. Since we don't really have any control over how the glTF is constructed, it should give us more flexibility to deal with these sorts of things if we're constructing the glTF (and potentially USD) files ourselves.

I've already got a basic working single-layer prototype of this for the volume exports (using marching cubes and constructing the glTF directly with gltflib). Should be relatively straightforward, I think, to extend this for the scatter as well.

ipyvolume compatibility

Compatibility of the existing export code with ipyvolume should be explored so that this can be used with glue-jupyter.

Support scatter layers in volume viewer

Something that's supported in the vispy volume viewer that our export doesn't currently support is the ability to add scatter layers. While this won't be hard to do in terms of constructing the file (we can call a dispatch function based on the layer state type to construct the appropriate meshes), it will require a bit of refactoring to the current UI in order to swap out the bottom piece of the dialog on the fly (as the different types of layers have different options).

Opacity binning in voxel export

While we don't make multiple materials with the exact same opacity for the voxel export, there will generally be a lot of close-but-different opacity values for these. This means that we're adding a lot of materials to the exported file, many of which probably aren't distinguishable. We should consider adding a voxel export option that allows a user to specify how many opacity bins/what resolution they want to have for opacity. For example, if a user set this value to 100, we would round to the nearest 0.01 when determining the final voxel opacity.

WWT 3D mode export

This might be a bit optimistic, but it seems to me that we could export point layers from WWT's 3D mode using the same basic principles as the scatter viewer. Converting from RA/Dec to 3D is easy, and then it's really just a scatter plot.

It may be that this isn't really worth it without the context of the Solar System/Milky Way/etc. Adding in those sorts of things seems trickier. But since this seems like it could be simple, I might give it a try.