ianmackenzie / elm-3d-scene Goto Github PK

Currently Geometry exists as a separate type so that a single mesh can be reused with different materials to reduce the amount of data stored on the GPU. However, in most cases I suspect the more important optimization will be to batch many pieces of geometry together into a single mesh (and therefore draw call), which will require the geometries to all have materials applied (that is, material parameters will have to be stored per vertex instead of via uniforms, so geometry and material can't be two separate types). Combining Geometry and Node into a single Drawable type also just generally simplifies the package.

To start:

• Planar (based on a sketch plane)
• Spherical (based on a frame)
• Cylindrical (based on a frame)

Would also need a way to specify the texture scale (meters per pixel? texture width? width and height?).

Should likely be implemented as functions that transform a Material - convert one that requires UV coordinates into one that doesn't.

Allows for things like

Block3d.centeredOn axes ( length, width, height )
    |> Scene3d.block Materials.aluminum

or

List.map (Scene3d.mesh Materials.aluminum) listOfMeshes

Much of the necessary functionality for supporting normal maps is already present inside elm-3d-scene, but there's a bit more thought to do on exactly how to expose it and make sure that all corner cases are correctly handled.

For example, some work remains on how exactly normal map textures should be represented/handled - try to fit it into the existing Texture framework as a Texture Direction3d or similar, or have a specialized NormalMap type? The latter is likely better since it would then be easier to support custom options such as whether normal map texture should be assumed to be in OpenGL or DirectX format.

In addition, normal mapping often involves a sign flag - proper support for this sign flag needs to be added and tested.

Add function that combines multiple meshes within a single Drawable into one big mesh, to reduce the number of total draw calls:

Drawable.pack : Drawable -> Drawable

Alternate names:

• compact
• merge
• flatten
• combine
• optimize
• bake
• compile
• collapse

This means that the original vertex data must be retained for each leaf Drawable, so that multiple transformed drawables can all be converted into the same coordinate system before being converted back into one big WebGL mesh. One efficient way to do this would be to store vertex data in 'fat objects' with types such as

{ x : Float
, y : Float
, z : Float
, nx : Float
, ny : Float
, nz : Float
, r : Float
, g : Float
, b : Float
, rg : Float -- roughness
, mt : Float -- metallic
}

which could then be stored and transformed efficiently.

Add support for using textures to define:

• Color
• Roughness
• Normal maps

Add rough support by controlling draw order, full support will likely involve depth peeling (multiple render passes)

Or at least line segments; likely use Unreal method

This is a bit of a placeholder issue since my current thinking is that support for different file formats should live in their own packages, so (e.g.) a COLLADA package can be focused on parsing COLLADA files into nice Elm data types and elm-3d-scene can remain focused on rendering. That said, one way or another there should definitely be an easy way to load models into elm-3d-scene. Potential file formats to support include:

• OBJ
• COLLADA (.dae)
• FBX
• glTF
• STL

Please add a comment if there's a particular file format you'd like to see supported!

Especially useful for doing things like normal mapping to give surfaces a more realistic slightly bumpy look, without needing an actual texture image

Problem
Files in the example folder don't compile.

Steps to reproduce

git clone [email protected]:ianmackenzie/elm-3d-scene.git
cd examples
npx elm make Triangles.elm

Possible Cause
Namespace clashes of tesk9/palette(used in the package) and avh4/elm-color(used in the example). Both export a module named Color.

Fix
I am willing to contribute a fix if that is helpful. I would need some feedback what the the best solution would be? I would propose to remove the the avh4/elm-color dependency. But it makes me wonder how this example ever worked? I thought dependencies with clashing module names don't ever work together because the Elm compiler cannot tell them apart.

Full Output

Detected problems in 2 modules.
-- MODULE NOT FOUND - /home/deedo/projects/experiments/elm-3d-scene/src/Scene3d.elm

You are trying to import a `Color.Transparent` module:

19| import Color.Transparent
           ^^^^^^^^^^^^^^^^^
I checked the "dependencies" and "source-directories" listed in your elm.json,
but I cannot find it! Maybe it is a typo for one of these names?

    Current
    Capacitance
    SolidAngle
    SubstanceAmount

Hint: If it is not a typo, check the "dependencies" and "source-directories" of
your elm.json to make sure all the packages you need are listed there!

                                                                  Scene3d  ↑    
====o======================================================================o====
    ↓  Scene3d.ColorConversions


-- NAMING ERROR - /home/deedo/projects/experiments/elm-3d-scene/src/Scene3d/ColorConversions.elm

I cannot find a `Color.fromRGB` variable:

40|     Color.fromRGB
        ^^^^^^^^^^^^^
The `Color` module does not expose a `fromRGB` variable. These names seem close
though:

    Color.fromRgba
    Color.fromHsla
    Color.rgb
    Color.toRgba

Hint: Read <https://elm-lang.org/0.19.1/imports> to see how `import`
declarations work in Elm.

-- NAMING ERROR - /home/deedo/projects/experiments/elm-3d-scene/src/Scene3d/ColorConversions.elm

I cannot find a `Color.toRGB` variable:

29|             Color.toRGB color
                ^^^^^^^^^^^
The `Color` module does not expose a `toRGB` variable. These names seem close
though:

    Color.toRgba
    Color.rgb
    Color.rgba
    Color.brown

Hint: Read <https://elm-lang.org/0.19.1/imports> to see how `import`
declarations work in Elm.

Possible renderers (could potentially support multiple ones!):

• Mitsuba
• YafaRay
• POV-Ray
• Appleseed
• pt
• Cycles (file format examples)
• Radiance
• MoonRay

Could either convert to a particular file format (like Mitsuba's XML format) and then render using a command-line tool, or encode as JSON and send to a server that decodes the JSON and then renders directly using a ray tracing library like pt.

Should be possible to have a sphere mesh where every vertex is just pair of U and V values, and the center point/radius are passed in as uniforms. Then the vertex shader could calculate vertex position, normal, tangent vector etc. (and you have texture coordinates for free!).

You could even share a given "UV grid" mesh among several different shapes as long as they have 2D parameterizations - spheres, cylindrical surfaces, toruses, spline surfaces...

Problem
elm-3d-scene makes use of bunch of other libraries. A couple of them are not namespaced for people (like me) who haven't used those before it is difficult to see where the imports come from.

import Angle -- ianmackenzie/elm-units
import Camera3d  -- ianmackenzie/camera-3d
import Color -- tesk9/palette
import Color.Transparent -- tesk9/palette
import Direction3d -- tesk9/palette
import Length -- ianmackenzie/elm-units
import Pixels exposing (Pixels, pixels) -- ianmackenzie/elm-units
import Point3d -- ianmackenzie/elm-geometry
import Scene3d exposing (noDirectLighting, noEnvironmentalLighting) -- ianmackenzie/elm-3d-scene
import Scene3d.Chromaticity as Chromaticity -- ianmackenzie/elm-3d-scene
import Scene3d.Drawable as Drawable -- ianmackenzie/elm-3d-scene
import Scene3d.Exposure as Exposure -- ianmackenzie/elm-3d-scene
import Scene3d.Mesh as Mesh -- ianmackenzie/elm-3d-scene
import Triangle3d -- ianmackenzie/elm-geometry
import Viewpoint3d -- ianmackenzie/camera-3d

Suggested Solution
Namespace the modules in the other libraries like Unit.Length instead of Lenght for elm-units. Other affected libraries are elm-geometry, elm-3d-camera and possibly more.

Other Implications
Namespacing would also help dodge the fact that the compiler currently cannot diambiguate modules with the same name. See here

Other
If there is a decision to implement what this issue suggests, it may be better to track on the respective repos of the libraries. But until them I felt it would give a better insight over the issue here.

The current Entity.shadowDrawFunction doesn't work properly when a shadow is mirrored, since that changes the handedness of all the faces. May need to add a uniform to the shader, or perhaps just flip the WebGL.Settings.cullFace setting if that's enough.

Set up default lighting, exposure, white balance, dynamic range etc.:

Scene3d.sunnyDay
    { sunlight : Direction3d coordinates
    , shadows : Bool
    , dimensions : ( Quantity Float Pixels, Quantity Float Pixels )
    , camera : Camera3d Meters coordinates
    , background : Background coordinates
    }
    -> List (Entity coordinates)
    -> Html msg

Scene3d.cloudyDay
    { dimensions : ( Quantity Float Pixels, Quantity Float Pixels )
    , camera : Camera3d Meters coordinates
    , background : Background coordinates
    }
    -> List (Entity coordinates)
    -> Html msg

Scene3d.office
    { directLighting : DirectLighting coordinates
    , dimensions : ( Quantity Float Pixels, Quantity Float Pixels )
    , camera : Camera3d Meters coordinates
    , background : Background coordinates
    }
    -> List (Entity coordinates)
    -> Html msg

Start with support for equirectangular panoramas, something like

Scene3d.texturedBackground
    { groundPlane : SketchPlane3d Meters coordinates
    , equirectangularPanorama : Material.Unlit
    }
    -> Background coordinates

Depends on #46.

Act as constant attenuation factor for ambient lighting, no effect on direct lighting

Would be useful to have a few representative scenes (standalone web apps) with a shape in the middle that has customizable material, roughness etc., and be able to tweak the scene lighting; should provide a nice way to quickly find reasonable material/lighting values

This would contain generated WebGL meshes at varying levels of detail, and would be updated in an app's update function based on a 3D scene, a camera position, a desired rendering accuracy (in pixels), and possibly a caching strategy (to determine when generated meshes should be discarded). The cache would then be used in view to actually render as WebGL.

This system would have many benefits:

• Can support dynamic levels of detail transparently for analytical shapes like spline curves, extruded/revolved surfaces etc: generate meshes with different levels of accuracy on the fly as needed
• Similarly, can separate construction of a scene from curves/surfaces from the decision of what accuracy to render at
• In some cases, meshing can be avoided entirely: for example, if the scene is actually encoded as JSON or in a particular file format to be rendered on a server and sent back as an image/streaming video etc.

Having a cache system would likely mean that individual drawables should have a pre-computed hash of their content, to be used as an ID for looking up corresponding meshes in the cache.

Overcast, sunny and partly overcast, as described in http://silviojemma.com/public/papers/lighting/spherical-harmonic-lighting.pdf.

For best results they should probably use something like importance sampling taking both the BRDF and lighting equation into account (heavily weight high-reflection directions, but also the direction sunlight is coming from...some sort of product?)

Scene3d.overlay : (Camera3d -> Drawing2d.Element msg) -> Option msg

used as

Scene3d.renderWith [ overlay ] lights camera scene

If multiple overlay options are provided, they can simply be combined into one drawing.

Allows for future API additions such as using ambient/environmental lighting as background, or a separate skybox texture

Currently Drawables must be constructed with a specific color/material, but it is possible to vary color dynamically without reloading the mesh data. This might be done by adding functions like

Drawable.trianglesWithDynamicColor : List Triangle3d -> Vec3 -> Drawable

which are designed to be partially applied, resulting in a Vec3 -> Drawable function where every call with a different color ends up reusing the same underlying mesh.

This could also be done by changing the argument order of existing functions to always pass color/material last, but that could interfere with some optimizations - for example, if a simple flat color is passed as the material for some mesh, the implementation can choose to store positions only and discard normal vectors. If the implementation had to always assume that materials will be changed dynamically (potentially to one where normals are required), then such optimizations would become harder or impossible.

See "Approximation" section of https://en.m.wikipedia.org/wiki/Planckian_locus

If using HTML overlays, try using transform: translateZ(0) to force GPU use

Likely start with Reinhard, but try out since of the more sophisticated methods from https://imdoingitwrong.wordpress.com/2010/08/19/why-reinhard-desaturates-my-blacks-3/

Perhaps something like

Drawable.withLevelsOfDetail
    [ { accuracy = millimeters 1, drawable = ... }
    , { accuracy = millimeters 5, drawable = ... }
    , { accuracy = millimeters 10, drawable = ... }
    ]

and then the correct drawable would be chosen based on the current camera position. Note that the near clip distance of the camera can be used as a lower bound for how close the object can be to the viewer, to help determine required accuracy in world units based on desired accuracy in pixels!

See http://www.mikktspace.com/, "Pixel Shader Transformation" section - use raw interpolated normal/tangent vectors to form bitangent vector, and normalize only the final result

Native WebGL line rendering is basically terrible - can't reliably do anything other than 1 pixel wide lines and even that doesn't always seem to work. It would be better to try to implement a triangle-based solution of some sort. Possibly useful resources:

• https://mattdesl.svbtle.com/drawing-lines-is-hard
• https://wwwtyro.net/2019/11/18/instanced-lines.html

Promising approach: configure Amazon Fargate with Mitsuba set up in a container image. When a rendering request comes in, spin up 10s/100s/1000s of containers with Mitsuba running on each one as a server, then start a networked render and tear down the cluster when finished. Perhaps accept donations if it gets popular, and/or throttle requests based on IP address or something...

Rough idea: have Elm app running on the same machine as a Rust executable doing high-performance rendering with Vulkan, real-time ray tracing etc.; stream current scene description from Elm to Rust, and stream back rendering result as video.

Should be able to reuse much of the shadow edges logic to tag lines with their adjacent face normals, then only render silhouette edges

Various possible approaches:

• Draw lines, apply polygon offset to faces (simplest but mediocre quality/performance)
• Fragment shader based approaches (relies on including barycentric coordinates or similar in vertex data, requires support for GLSL standard derivatives)
• Post processing of depth map data (requires support for multipass rendering)

Use linearly transformed cosines: https://eheitzresearch.wordpress.com/415-2/

The current shadow implementation uses the simpler-to-understand but naive implementation that doesn't work if the viewer is inside the shadow volume; should be updated to the more sophisticated (and more efficient) implementation instead

Perhaps using http://evanw.github.io/glslx/ or a similar tool

Big and complex topic; could include

• Image-based (cubemap) lighting
• Specialized hemispherical or similar lighting
• Importance sampling vs. texture pre-processing
• HDR textures

Potentially useful technique for normal vector transformation: https://twitter.com/zeuxcg/status/1226334381091872769?s=19

Either using traditional shadow maps or perhaps something clever/specialized like sets of analytical occluders (spheres, boxes etc.).

One cool approach would be to dynamically generate GLSL code for a given set of occluders; this could include hardcoded R-tree-like culling logic to quickly determine which occluders may affect a given light/ray.

Would more easily allow for float textures to specify which color channel should be used (or if average/greyscale value should be used instead)

The WebGL.clearColor option is only applied once (when the scene is initially rendered), meaning that it's not possible to have a dynamic background color using WebGL.clearColor alone. However, it should be possible for elm-3d-scene to work around this limitation by:

• always setting the clear color to be fully transparent
• rendering (as the very first entity) a full-screen quad using the actual specified background color, without touching the depth buffer
• and then rendering all other entities on top of that

https://github.com/donmccurdy/msdf-bmfont-web
https://github.com/Chlumsky/msdfgen
https://github.com/Chlumsky/msdf-atlas-gen
https://news.ycombinator.com/item?id=20020664

Many different approaches - research needed to be done on which ones are most easily implementable, most performant, most physically defensible, actually possible using WebGL and Elm's implementation of it, etc.

As discussed in the comments on dc83017, it would be nice to split out the C++ texture-generation code into its own repository. This would avoid this repository from showing up on GitHub as mostly C++ (which might make people think the package is based on C++ compiled to WebAssembly or something).

elm-3d-scene currently uses the simple sRGB gamma correction approximation of using an exponent of 2.2 (or 1/2.2); this should be changed (in both shaders and conversion of Color values to linear colors) to use the actual algorithm.

Important for realistic soft lighting; likely use the Unreal method

Show things like:

• triangle edges
• curve vertices
• normal vectors
• tangent vectors
• UV grid

Enable per object (can add to a group to recursively enable debugging on all members)

Handy for debugging but also for generating cool-looking demo images =)

Do a pass through the docs and point out where types/modules come from other packages; may also be worth adding a section to the README like "Relationship to other packages" or similar.

Will require new shaders:

• Water
• Ceramics (very rough/retroreflective materials)
• Blinn-Phong
• Lambertian (diffuse)
• Skin (subsurface scattering)
  • Simple approximation once normal maps are supported: use normal map for specular lighting
    but not for diffuse lighting
• Thin translucent materials like leaves and lamp shades that 'glow' from transmitted light