Code Monkey home page Code Monkey logo

xc3_lib's Introduction

xc3_lib GitHub release (latest by date including pre-releases)

Rust libraries and tools for working with rendering related file formats for Xenoblade Chronicles X, Xenoblade Chronicles 1 DE, Xenoblade Chronicles 2, and Xenoblade Chronicles 3.

Report any bugs or request new features in issues. Download precompiled binaries for the tools in releases. Python bindings for xc3_model are available with xc3_model_py. See the xenoblade rendering research website for information on topics related to in game rendering.

Formats

xc3_lib supports a number of in game formats. All formats support reading. Write support is still a WIP for some formats. Click on the links to open the corresponding Rust module in xc3_lib. Extensions starting with wi are for the Switch like wimdo or wismt. Extensions starting with pc are for PC builds like pcmdo or pcsmt. Extensions starting with ca are for the Wii U like camdo or casmt.

Format Magic Extensions Write
Apmd DMPA wimdo ✔️
Bc BC anm, motstm_data ✔️*
Beb beb ✔️
Beh hdev beh
Bmn BMN bmn
Dhal LAHD wilay ✔️*
Eva eva eva ✔️*
Lagp LAGP wilay ✔️*
Laps LAPS wilay ✔️*
Ltpc LTPC ✔️
Mibl LBIM witex, witx ✔️
Msmd DMSM wismhd
Msrd DRSM wismt ✔️*
Mtxt MTXT catex, calut, caavp ✔️
Mxmd DMXM wimdo ✔️*
MxmdLegacy MXMD camdo
Sar1 1RAS arc, chr, mot ✔️
Spch HCPS wishp ✔️
Xbc1 xbc1 wismt ✔️

* Some files are not binary identical with the originals after saving.

Projects

See Architecture for a design overview of the various projects. Click on the docs.rs links below to see the generated rustdoc documentation.

Libraries

  • Crates.io docs.rs - file format library
  • Crates.io docs.rs - higher level API for xc3_lib
  • Crates.io docs.rs - model and map renderer
  • Crates.io docs.rs - binary writing and layout

Binaries

  • xc3_gltf - convert models and maps to glTF
  • xc3_test - test against files in an extracted dump
  • xc3_tex - convert textures to and from common formats and replace textures in wilay and wimdo files
  • xc3_viewer - simple model viewer for testing xc3_wgpu
  • xc3_wgpu_batch - batch render models and maps to PNG

Usage

These projects are still highly unstable. When using the latest version from github, specify a specific git revision or commit the Cargo.lock file to source control. This locks the version and avoids any breaking changes. The debug or JSON output has not stabilized and should not be assumed to be the same between commits.

xc3_model = { git = "https://github.com/ScanMountGoat/xc3_lib", rev = "commit_hash" }
xc3_wgpu = { git = "https://github.com/ScanMountGoat/xc3_lib", rev = "commit_hash" }
xc3_lib = { git = "https://github.com/ScanMountGoat/xc3_lib", rev = "commit_hash" }

Building

After installing the Rust toolchain, run cargo build --release in the repository directory to build the tools to target/release. Running cargo build without the --release will result in faster compile times during development but dramatically worse runtime performance. The tools can also be run using cargo run --release -p <project> <args>. xc3_tex uses image_dds, which supports Windows x86, Linux x86, MacOS x86, and MacOS Apple Silicon due to using precompiled kernels for DDS encoding. Other projects should build on other platforms without issues.

Credits

This project is based on previous reverse engineering work, including work done for Xenoblade X and Xenoblade 2. Special thanks go to members of the World Tree Research discord (formerly the World of Alrest discord) for their assistance.

xc3_lib's People

Contributors

roccodev avatar scanmountgoat avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

xc3_lib's Issues

Xenoblade X shader database

This requires finding a shader decompiler that's easy to build and use as an external tool similar to Ryujinx.ShaderTools. xc3_shader will need additional tests to make sure that the GLSL syntax works with the existing analysis code.

emissive texture support

This requires additional research to differentiate between emission and hair specular that both use the output texture at index 5.

  • gltf
  • xc3_wgpu

Feature Request: Ability to Edit a Model's Materials

When replacing certain model textures, such as Mythra's hair textures, the in-game shader effects will "blend" the color of the new texture with the color it was originally designed to accent. The ability to edit these materials should hopefully improve the look of certain texture replacements in-game.

Material Shaders for Massive Melee Mythra's Hair
Untitled1

Massive Melee Mythra's (Right) hair texture replaced with with Pyra-Style Mythra's (Left)
Untitled2

Sword Fighter Class jacket textures replaced with Noah's Color 2 jacket textures (Noah's Color 2 Outfit used for Comparison)
Untitled3

vertex attribute data type 9 not supported

Getting this error when attempting to load DE's Colony 9(ma0101) & Bionis Leg(ma0301). Seems to be an issue affecting only certain maps.

0: Error: no variants matched at 0x1298
           Unexpected value for enum: 9
 1: While parsing field 'data_type' in VertexAttribute
     at xc3_lib\src\vertex.rs:95
 2: While parsing field 'attributes' in VertexBufferDescriptor
     at xc3_lib\src\vertex.rs:82
 3: While parsing field 'vertex_buffers' in VertexData
     at xc3_lib\src\vertex.rs:21

Extend xc3_tex to include wilay (LAHD) support

The xc3_lib appears to already support reading and writing wilay (LAHD) textures. Extending the xc3_tex programme to include this support for this file format would enable users to more easily modify menu/UI images.

Currently I am unaware of any other programme that enables users to generate wilay textures - only ones that write them out to png (eg. XbTool). Therefore, there remains a need for someone to make such support accessible to modders.

xc3_tex Texture Replacements with v2.2.0 Leisure Time models fail to build correctly

Not sure if it's because xc3_lib hasn't been updated since v2.2.0 dropped, but I've been unable to properly replace the textures of the Leisure Time models (tested only with Mio and Sena's so far). Loading the edited models in-game via Layered FS causes the game to freeze, and Noesis gives the following error when it tries to preview them (though xc3_viewer seems to have no problems reading/previewing them).
error

Wilay images used as 'sprite sheets/image maps' don't appear to output correctly

NB. I'm using Xenoblade Chronicles 2 as my example as this appears to be the easiest to test and handles the problem cleaner. On XC3 the issue may crash the game

The addition of .wilay support to xc3_tex has enabled the successful export and re-import of various images in the 'menu\images' directory (thank you!). Examples include the thumbnails in the events theatre or the blade character images - these successfully show up in game.
However, wilay files which contain a single image with multiple icons or images drawn on it fail to display in-game when re-imported. Examples include character portrait image sheet or button icons layout sheet (I've used the buttons as an example image below as the character sheet contains spoilers for some people).
button_0

I have tested all of the above with unedited dds images output by xc3_tex - ie. I have tried re-importing the exact image that xc3_tex has outputted without modifying it in anyway. This should rule out any issue that could be created by the user modifying the image and re-saving.

In the example image below, you can see that a modified mythra (fim_bl_002.wilay) is displaying successfully, while character portraits in the top right (mnu_facel.wilay) and button icons in the bottom right (button.wilay) are failing to show:
xc2_mmmytha_noicons

xc3_tex bad magic error for xc2 LAPS wilay

Wilay images located in 'menu\layout' fail to process with the following error:

thread 'main' panicked at xc3_tex\src\main.rs:79:76:
called `Result::unwrap()` on an `Err` value: bad magic at 0x0: [76, 65, 80, 83]

Ideally adding support for whatever is different about these files would be nice, though if this is not a short term fix then perhaps a cleaner error message could be helpful so people understand that these files are different and as yet unsupported, even though they share the same file extension.

xeno1/chr/np/np860109.wimdo freezes in game after editing with xc3_tex

Not sure if this is an issue specific to XCDE Alvis' model (np860109), or if it extends to other NPC models as well, but editing any of Alvis' textures appears to cause the game to freeze upon the edited model's load. The .wimdo and .wismt don't appear corrupted, as they load just fine in Noesis, but I still assume something went wrong. Below is the main texture I'm trying to edit in:

np860109.16.NP860101_NEC_COL
image

Expected 2D texture but found dimension D3

I know xc1 isn't officially supported by wgpu_batch, but here's an error that shows up frequently when processing xb1 enemies.
examples: en150112, en150106, en150101
ERROR [xc3_wgpu::material] Expected 2D texture but found dimension D3.

xc3_tex xc2/menu/image/mnu032_button_push_main.wilay fails to rebuild properly

Some wilay files contain multiple images (eg. mnu032_button_push_main.wilay). When these files are re-compiled some of the images contained within the wilay appear offset when displayed onscreen, while others may be offset outside the visible screen.

Here are the contents of the mnu032_button_push_main.wilay when extracted:
push_button_map
The first image is the background, the second is a glow around the background, followed by pairs of buttons for swapping between the non-depressed and depressed states.

The original file should show these all layered on top of each other as below (ie. background, ring around border, 5th image [non-depressed B]):
xc2-button-no_shift

The re-compiled file shows the first image in the correct position (the background) but is then failing to show the ring around border on-screen (it may be offset somewhere outside the visible screen?) then shows the fourth image (depressed button A) offset from the first. It also appears to fail to toggle between depressed and non-depressed states - possibly because the actual button that should be shown on screen are the 'B' pair which may be offset somewhere else...
xc2-button-shift

I can confirm that this still happens even when I do not edit or reorder any of the images, but simply extract, then immediately recompile the wilay without any change.

DE Broken menu textures

Tested on v0.7.0

img0103_1.0, img0103_2.0, img0103_3.0
img0103_1 0

mf03_map01_ma2401_f01_map.0
mf03_map01_ma2401_f01_map 0

npg_photo14.0
npg_photo14 0

Load multiple wimdo and camdo models

Some models are split across multiple files. This applies to all supported games. One approach is to add model loading functions that take an iterator of paths. Loading a single model can call this function internally with std::iter::once. Merging a ModelRoot should also merge the skeletons. Skeleton merging can be tested with unit tests.

  • load_models and load_models_legacy
  • merge ModelRoot
  • support multiple files for xc3_gltf and xc3_viewer

Add support for beb and beh cutscene files for Xenoblade 3

Read/write support for xc3_lib for the event/*.beb and event/*.beh formats. The basic steps are outlined below. Please comment if you plan on working on this or need additional help. Cutscenes aren't something I'm prioritizing, so I may not get around to this for a while.

  • add src/beb.rs with a Beb struct and src/beh.rs with a Beh struct to xc3_lib
  • create structs for the file formats using the other modules as a reference (unk fields are fine)
  • add beb::Beb and beh::Beh to file_read_impl and file_write_full_impl in lib.rs to add read/write helpers
  • test the reading and writing in a separate CLI project or by adding a main.rs file (please don't commit this)
  • add test cases to xc3_test for testing read/write for all files (optional)

Useful Links:
https://github.com/ScanMountGoat/xc3_lib/blob/main/DEVELOPMENT.md
https://docs.rs/binrw/latest/binrw/
https://github.com/ScanMountGoat/xc3_lib/blob/main/Offsets.md

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.