Code Monkey home page Code Monkey logo

opengametools's Introduction

open game tools

Open game tools is a set of unencumbered, free, lightweight, easy-to-integrate tools for use in game development.

So far it contains these libraries:

... and these applications:

  • vox2fbx.cpp an application to extract models from .vox and save them to ascii fbx
  • vox2obj.cpp an application to extract full frames from `.vox' and save them to Wavefront OBJ files. This tool also supports writing all frames out to individual single-frame .vox files.
  • voxseparate.cpp an application to extract models from .vox and save them to separate .vox files
  • voxmerge.cpp an application to merge multiple .vox files into a single .vox file

Please consider contributing fixes, extensions, bug reports or feature requests to this project. If you have example scenes that fail to load or save correctly, or have additional issues, feel free to file an issue on github and I'd be happy to investigate and make fixes when I have the time.

See CONTRIBUTING.md for more details.

ogt_vox: MagicaVoxel scene reader, writer and merger

A C++ reader, writer and scene merger for MagicaVoxel's vox file format.

Scene reading (.vox)

With this library, reading a .vox file will result in a ogt_vox_scene structure which you use to:

  • enumerate all or a subset of instance placements within a .vox file
  • get the transforms for those instances as a matrix flattened relative to the scene file, or relative to their group hierarchy
  • access the voxel model referenced by those placements ie. the 3d color grid
  • access the palette colors and materials
  • access layer, group and visible/hidden state for instances / groups
  • access model-swap keyframes for instances
  • access transform keyframes for instances / groups
  • access the camera settings

I use this library to deep-reference models in .vox files and turn them into triangle meshes in my engine's importer/cooker. I also use it in an in-engine wizard to import all instance/scene data into your own engine scene format. This can allow your artists to use MagicaVoxel as a level editor, or simply a tool for managing a kit, palette of module-set to be used within levels or objects within your own editor.

Scene Writing (.vox)

The C++ writer for MagicaVoxel's vox file format takes a user-constructed ogt_vox_scene structure and serializes it to .vox file format. When saved out to disk, it'll be loadable in MagicaVoxel.

My own testing involved going through all of my on-disk vox files, and verified they load in MagicaVoxel 0.99.3-alpha. The files are usually smaller than the source .vox though because they will not preserve all the chunks within the original .vox file.

Scene Merging (.vox)

You can load multiple .vox files simultaneously to obtain multiple ogt_vox_scene structures. You can then merge them into a single ogt_vox_scene and save it out.

If they have different palettes, the merger will do closest-color matches to try preserve the original colors in the output file.

If you find yourself needing to combine multiple files with very similar but non-identical palettes then this is probably the tool for you. If you need to combine multiple files with completely different palettes that have only a few entries used, then this may also be for you!

It should always work, but it may not give a good color fit if the voxel files you use most of their colors and have their own very different palettes. If you want to simply replace the merged file with a very specific palette, you can provide that to the merge function too.

In the future, I might add a more holistic algorithm that prioritizes allocation to the final palette of colors based on their distance from other colors.

Usage

See demo_vox.cpp for a simple example.

  1. Include ogt_vox.h directly from one of your C or CPP files:

         #define OGT_VOX_IMPLEMENTATION
         #include "ogt_vox.h"
  2. Use your own file utilities to load a .vox file into a memory buffer. eg.

     FILE* fp = fopen("file.vox", "rb");
     uint32_t buffer_size = _filelength(_fileno(fp));
     uint8_t* buffer = new uint8_t[buffer_size];
     fread(buffer, buffer_size, 1, fp);
     fclose(fp);
  3. Pass the buffer into ogt_vox_read_scene to get a scene object eg.

     const ogt_vox_scene* scene = ogt_vox_read_scene(buffer, buffer_size);
     // the buffer can be safely deleted once the scene is instantiated.
     delete[] buffer;
  4. Now read all the information you need from the scene object and use it to import voxel data/instances. eg. Here, I just print some basic information:

     printf("# models: %u\n",  scene->num_models);
     printf("# instances: %u\n", scene->num_instances);
  5. Shut down the scene to release the memory it allocated. eg.

    ogt_vox_destroy_scene(scene);

Note

The reader, writer and merge functionaliy supports the most relevant chunks as described here:

MATL/MATT are not yet merged, but it is expected to come in a later change. CAVEAT: loading of MATL chunks are most accurate when written by recent Magicavoxel versions (0.99.6.2 and 0.99.6.3 at time of writing). Chunks written by earlier versions may not be decoded accurately.

rOBJ chunks are not supported in any capacity for now. If there is interest, do let us know in the Issues section.

ogt_voxel_meshify: converts voxel grid data to triangle mesh data

This module contains 3 routines for converting paletted voxel grid data to triangle mesh data.

  • ogt_mesh_from_paletted_voxels_simple: returns the most naieve mesh possible, which will be tessellated at voxel granularity. image

  • ogt_mesh_from_paletted_voxels_greedy: uses a greedy box-expansion pass to replce rectangular regions of voxels of the same color with a larger polygon that covers the box. It will generally produce t-junctions which can make rasterization not water-tight based on your camera/project/distances. image

  • ogt_mesh_from_paletted_voxels_polygon: will polygonize and triangulate connected voxels that are of the same color. The boundary of the polygon will be tessellated only to the degree that is necessary to there are tessellations at color discontinuities. Generally produces slightly more triangles that greedy meshing but it is mostly water-tight, except for a very small number of cases. image

More documentation and a demo will be added soon.

vox2fbx: Tool for extracting meshifying models from Magicavoxel .vox and saving them to .fbx

This is a command-line tool which will find all named voxel grid models in a vox file, turn them into a mesh using one of 3 algorithms (simple, greedy or polygon) each of which produce different density mesh with varying levels of water-tightness (ie. t-junctions), then save those meshes out to individual fbx files.

It can be used from command-line with these args:

  • --mesh_algorithm <algo> : (default: polygon) sets the meshing mode where is one of: simple, greedy or polygon
  • --named-models-only : (default: disabled) will only generate an fbx for models in the vox file that have named instances
  • --y-as-up : (default: disabled) rotate model on export so that y is up

It can also be used from windows explorer by dragging-and-drop your .vox files onto it, and it will produce an output fbx file for each grid model in each .vox file.

vox2obj: Tool for baking entire scenes in a Magicavoxel .vox into a single mesh and saving them to .obj

This is a command-line tool which will find all named voxel grid models in a vox file, turn them into a single mesh using one of 3 algorithms (simple, greedy or polygon) each of which produce different density mesh with varying levels of water-tightness (ie. t-junctions), then save out all frames or a specific set of animation frames out to a .obj file. Each animation frame can optionally be exported into the same .obj file as separate subobjects.

It can be used from command-line with these args:

  • --mesh_algorithm <algo> : (default: polygon) sets the meshing mode where is one of: simple, greedy or polygon
  • --all_frames_in_one : (default: disabled) specifies that all frames should be written into a single output file
  • --output_name <name> : (default: disabled): name of output files
  • --scale <value> : (default: 1.0): scaling factor to apply to output voxels
  • --frames <first> <last> : which frame range to extract. If not specified, will extract all frames between min and max keyframes within the .vox file.

It can also be used from windows explorer by dragging-and-drop your .vox files onto it, and it will produce an output .obj file with the above defaults.

voxseparate: Tool for extracting models from a multi-model Magicavoxel .vox and saving them to individual .vox files

Provide input .vox files on the command line, and output files will be written out to the same directory, with autogenerated names based on the names of instances of models within these input .vox files.

It can also be used from windows explorer by dragging-and-drop your .vox files onto it, and it will produce an output vox file for each model in each .vox file.

voxmerge: Tool for merging multiple Magicavoxel .vox files into a single .vox file.

Provide an output filename, and one-or-more multiple input .vox files and the merged scene will be written out. The output file will do a basic best fit of palettes colors across all input .vox files and will emit only the used colors from all input vox files into the output file.

Projects using open game tools

Acknowledgements

Thanks to @ephtracy. So much would not have been possible without your generosity in releasing the tool free to the public, releasing information about the file format and patiently answering my DMs on twitter.

Thanks to @nothings for sharing your single-headers as well as the philosophy behind them. The latter has been invaluable to my personal growth.

Thanks to @ndreca_com for feedback on the ogt_vox library!

Thanks to all contributors so far!

opengametools's People

Contributors

dougbinks avatar groud avatar jibjoub avatar jorgenpt avatar jpaver avatar lllida avatar mgerhardy avatar mrpossoms avatar phyronnaz avatar wouterweynants avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

opengametools's Issues

ogt_vox: Compiler warnings in release mode

/vengi/src/modules/voxelformat/external/ogt_vox.h: In function ‘void generate_instances_for_node(_vox_array<const _vox_scene_node_*>&, const _vox_array<_vox_scene_node_>&, uint32_t, const _vox_array<unsigned int>&, const _vox_array<ogt_vox_model*>&, _vox_array<ogt_vox_instance>&, _vox_array<char>&, _vox_array<ogt_vox_group>&, uint32_t, bool)’:
/vengi/src/modules/voxelformat/external/ogt_vox.h:1027:45: warning: unused variable ‘last_group’ [-Wunused-variable]
 1027 |                     const _vox_scene_node_* last_group     = stack.peek_back(1);
      |                                             ^~~~~~~~~~
/vengi/src/modules/voxelformat/external/ogt_vox.h: In function ‘uint8_t* ogt_vox_write_scene(const ogt_vox_scene*, uint32_t*)’:
/vengi/src/modules/voxelformat/external/ogt_vox.h:2476:107: warning: the address of ‘color_string’ will never be NULL [-Waddress]
 2476 |             uint32_t layer_dict_keyvalue_count = (layer_name_string ? 1 : 0) + (hidden_string ? 1 : 0) + (color_string ? 1 : 0);
      |                                                                                                           ^~~~~~~~~~~~
/vengi/src/modules/voxelformat/external/ogt_vox.h:2472:18: note: ‘color_string’ declared here
 2472 |             char color_string[64];
      |                  ^~~~~~~~~~~~
[ 54%] Built target attrib
/vengi/src/modules/voxelformat/external/ogt_vox.h: In function ‘void ogt_vox_test()’:
/vengi/src/modules/voxelformat/external/ogt_vox.h:2890:25: warning: unused variable ‘test_message’ [-Wunused-variable]
 2890 |             const char* test_message = "failed compute_looped_frame_index test";
      |   

Objects Ids

Thank you very much for this library is it the perfect library to directly interact with vox files. Like you said it will enable us to use magicavoxel directly as a level editor. But I have a question about that. Since MagicaVoxel does not have a way to define attributes for instances (like game specific properties e.g. health, power, etc...). So we need to be able to have a stable id for object instances coming from vox files. Because we will have our custom scene format which will point to these objects coming from the vox files, and map our game specific properties to these ids. Do you have any recommendations? Do we just handle it manually (by using combination of name and instance id)? is model_id useful in this case?

Thanks

ogt_vox: group references broken by commit: flatten animations ...

Commit 137ff3d breaks the handling of group references.

Steps to reproduce:

  1. Create a scene in MV with several models
  2. Group the models
  3. Create a reference (SHIFT+move or use icon) and move the new reference somewhere.
  4. Open with ogt_vox before/after this commit. Scene is incorrect after this commit.

Example vox file: https://drive.google.com/file/d/1qpp7WUnk7_w02dCvjI0V3VTLA332dkWq/view?usp=sharing

EDIT: note that for me this fix isn't urgent as I'm able to use the previous commit plus cherry picked resize perf fix as I don't support animations for now.

ogt_vox: group instances are not preserved when writing scenes

I recently learned Magicavoxel supports copying groups by reference, which means the underlying scene hierarchy is a directed-acyclic-graph that has leaf nodes or group nodes with multiple parents ie. diamond patterns in the graph.

ogt_vox exposes the scene hierarchy as a tree with instances as the leaf nodes and with groups as the branches, so has no diamond patterns.

ogt_vox_read_scene does in fact work when loading scenes that contain group references, it just does not preserve the group information when the scene is re-saved out via ogt_vox_write_scene.

We should look into options for supporting this, and whether it can be done without changes to client code that uses ogt_vox.

ogt_vox: segfault when accessing invalid group indices

for (uint32_t i = 0; i < scene->num_instances; ++i) {
  ogt_assert(scene->instances[i].group_index == k_invalid_group_index || scene->instances[i].group_index < scene->num_groups, "Invalid group index");
}

I've added this to the end of ogt_vox_read_scene_with_flags for the attached vox file - and it triggers. There is an invalid group index loaded. Maybe we should add some sanity checks to the end of this function to ensure that the client code can be safe in accessing members via index.

vox2obj crashes on that input file, too, Magicavoxel is able to load it

magicavoxel.vox.zip

ogt_vox : materials are not merged correctly

Currently, the materials are well loaded with load_vox_scene but are not merged correctly in ogt_vox_merge_scenes. Thus no materials other than diffuse (default) exist in the merged scene.

ogt_vox: getting access to the dict key/value entries

Getting access to the key/value entries of every dict in a scene would be nice - this would e.g. allow to do something with the rOBJ chunks, but at least write them back properly to the file (because most of the are unknown or different from version to version anyway)

ogt_vox: expose animation looping status

This appears to exist as _loop=1 on nTRN and nSHP chunk node dicts, so should be read/merged/written on both groups (transform anims) and instances (transform and model animations).

ogt_vox: Hints about applying the transform

Maybe it's obvious for everybody except me, but I thought that sharing my current implementation might help others and is a good opportunity to get some feedback on my solution:

const glm::vec4 pivot(
	floor(model->size_x / 2.0f),
	floor(model->size_y / 2.0f),
	floor(model->size_z / 2.0f), 
	0.0f
);
/**
 * @brief Calculate the scene graph object transformation. Used for the voxel and the AABB of the volume.
 *
 * @param mat The world space model matrix (rotation and translation) for the chunk
 * @param pos The position inside the untransformed chunk (local position)
 * @param pivot The pivot to do the rotation around. This is the @code chunk_size - 1 + 0.5 @endcode. Please
 * note that the @c w component must be @c 0.0
 * @return glm::vec4 The transformed world position
 */
static inline glm::vec4 transform(const glm::mat4x4 &mat, const glm::ivec3 &pos, const glm::vec4 &pivot) {
	return glm::floor(mat * (glm::vec4((float)pos.x + 0.5f, (float)pos.y + 0.5f, (float)pos.z + 0.5f, 1.0f) - pivot));
}
const uint8_t *ogtVoxels = model->voxel_data
for (uint32_t k = 0; k < model->size_z; ++k) {
	for (uint32_t j = 0; j < model->size_y; ++j) {
		for (uint32_t i = 0; i < model->size_x; ++i, ++ogtVoxel) {
			if (ogtVoxel[0] == 0) {
				continue;
			}
			const voxel::Voxel voxel = voxel::createVoxel(voxel::VoxelType::Generic, _palette[ogtVoxel[0]]);
			const glm::ivec3& pos = transform(ogtMat, glm::ivec3(i, j, k), pivot);
			const glm::ivec3& poszUp = transform(zUpMat, pos, glm::ivec4(0));
			v->setVoxel(poszUp, voxel);
		}
	}
}

the zUpMat is just to bring everything into my own coordinate system (y upwards)

const glm::mat4 zUpMat = glm::rotate(glm::rotate(glm::mat4(1.0f), glm::radians(-90.0f), glm::vec3(1.0f, 0.0f, 0.0f)), glm::radians(180.0f), glm::vec3(0.0f, 0.0f, 1.0f));

If I am doing something stupid here, please let me know. If not it might help others.

Find the full source code at https://github.com/mgerhardy/vengi/blob/master/src/modules/voxelformat/VoxFormat.cpp

ogt_vox : add formal test cases

  • to prevent regressions with further dev, we'd like coverage on:
    -- files produced by various magicavoxel versions. eg. early versions that don't produce nSHP/nTRN/nGRP chunks.
    -- groups vs flattening
    -- animation sampling
  • we may also want to try some fuzzer testing.

ogt_voxel_meshify: Suppress geometry between chunks

For meshing minecraft style chunks, the current algorithms will always generate polygons on chunk boundaries as they assume the data outside the grid is non-solid.

It may be useful to add support for avoiding these polygons.

One possible way would be if user tries to meshify MxNxK voxels, but provides M+2xN+2xK+2 voxel grid that contain the neighbor cells within the outer-shell. It'd be simple enough to extend the meshifiers to only consider the inner MxNxK grid for meshing, but use the data on the shell for determining whether faces are required.

Any writing of voxels object to Wavefront obj?

Hello,

Great project! I looked at the source code and did not find a way to write the voxels into a mesh OBJ file. Is that correct? I know Magikavoxel can do it but I want to automate this. Is this hard to implement from the current state of the project? I never really worked with it but if I have time maybe I can have a look at it? Just a bit busy right now.

Best,
Quentin

ogt_vox: setting k_read_scene_flags_groups false looses hidden and layer information

If k_read_scene_flags_groups is set to false loading the scene will loose information about hidden groups or group layers & thus groups which are in hidden layers.

This is because in addition to flattening the transform hierarchy the instance group_index is set to 0.

I am not sure of the best way to resolve this. Preserving the group_index might work but would potentially break some code, flattening the hidden information might work but a group can be in a layer different to it's children so that also needs to be taken into account. A new flag might be needed (perhaps k_flatten_hidden which flattens both layer and group hidden).

For my own code I am now going to flatten the transform hierarchy myself to work around this.

ogt_vox: instance order differences between file versions

In MV vox files the instance order is important for the correct rendering of overlapping voxels. The first instance will 'replace' voxels in later instances. This instance order applies to groups and within groups.

Currently it seems that the instance order read by ogt_vox for version 200 files is inverted with respect to version 150 files. I haven't yet looked into whether this is ogt_vox or the actual file order.

My quick fix was to add a read for the version and switch order (extra note: it could be useful if the version was stored as part of the scene). I will look into this issue in more depth soon.

ogt_vox: material property _media_type not handled yet

It looks like there are some new or unhandled material properties for blend, glass and cloud materials

material prop possible values
_media_type _scatter, _emit, _sss (default is absorb - if _media_type is not available)
value desc
_sss Subsurface Scattering Media
_scatter Scatter Media
_emit Emissive Media
// media type for blend, glass and cloud materials
enum ogt_media_type {
  ogt_media_type_absorb,  // Absorb media
  ogt_media_type_scatter, // Scatter media
  ogt_media_type_emit,    // Emissive media
  ogt_media_type_sss,     // Subsurface scattering media
};

Example files:

(I still wonder whether it makes sense to give low level access to the dict data in the scene to support future values)

Retrieving the palette index from the mesh generator

Hello, thanks for this awesome library!

While working on using this library to import .vox models into Godot, it seems like I faced a limitation regarding how meshes are returned to the user. While using ogt_mesh_from_paletted_voxels_polygon I figured out I could not find a way to access the palette index associated to a given generated vertex. However, it would be very useful to me to retrieve the corresponding material properties (metalness, emissive, etc...).

Please let me know if I am right here, and there indeed no easy way to retrieve this. If I am and you are interested, I would be happy to contribute a simple enhancement to the library, likely by adding a palette index property in ogt_mesh_vertex (Or maybe removing the color property, since it could be retrieved from the palette anyway ?).

Feature request: Merging models

I'm hoping it would be an easy task to merge multiple models into a single model. I'm looking for functionality similar to MagicaVoxel's 'union' (U) function when selecting a root node.

Crashes due to unaligned memory access

/home/mgerhardy/dev/engine/src/modules/voxelformat/external/ogt_vox.h:1453:55: runtime error: member access within misaligned address 0x619000005581 for type 'struct ogt_vox_keyframe_transform', which requires 4 byte alignment
0x619000005581: note: pointer points here
 00 00 00  58 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00 00

this is due to the fact that the misc_data allocator is configured for char type - which is a 1 byte alignment - but the struct ogt_vox_keyframe_transform is of course not aligned to 1 byte.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.