puxtril / warframe-exporter Goto Github PK
View Code? Open in Web Editor NEWConverts Warframe's custom file formats into standard formats
Converts Warframe's custom file formats into standard formats
Open-world areas have (in addition to many others) their own model format for landscapes. Inside the cache, they often have the .lndscp file extension, but they always have the Enum value 42 (as of writing this). I have not taken a close look at these, but these would be very good to support full level extraction.
Examples provided by the --print-enums
flag:
/Lotus/Levels/InfestedMicroplanet/InfFleshscapeA/2.lndscp
/Lotus/Levels/InfestedMicroplanet/InfFleshscapeA/12.lndscp
/Lotus/Levels/Duviri/WarframeArenas/WFArenasEchoesOfDuviri/FractalPasturesCombat/0.lndscp
/Lotus/Levels/Duviri/MainlandAEast/0.lndscp
/Lotus/Levels/TheNewWar2021/Part1/CetusFire/0.lndscp
/Lotus/Levels/TheNewWar2021/VenusLandscapePostWar/0.lndscp
/Lotus/Levels/TheNewWar2021/CetusPostWar/0.lndscp
/Lotus/Levels/TheNewWar2021/PlainsLandscapePostWar/0.lndscp
/Lotus/Levels/NightWave/TheGlassMaker/CetusCrimeScene/0.lndscp
/Lotus/Levels/CivilianHubs/CetusRemaster/0.lndscp
Lightmap also has textures, which has not been known to contain useful textures until recently. I've discovered 2 new compression formats alongside the usual BC1-7 and 1 uncompressed format.
12 /Lotus/Levels/CorpusGasCityRemaster/GasExit01/LevelCapture_mm.png
30 /Lotus/Levels/ClanDojo/ClanDojoDuviriDeadEnd/Landscape0/T0u0.lchm
Hopefully in the near future until the animation is exported
Currently, the texture resolution needs to be calculated dynamically. Warframe provides a resolution, but this should only be used for the texture resolution ratio. The formula to calculate the actual resolution is a bit clunky.
Thanks to a discovery by @sehnryr (detailed in sehnryr/wfcache-api#1), this will be a lot simpler. Not a huge priority as the current implementation is fast enough and works without issue.
Textures extracted pre-ensmallening have a different compression in the cache files and are processed differently as textures. In the cache, they use LZ decompression. As textures, they need to be unswizzled before written to disk. Textures pre-ensmallening are missing data at the bottom - There's a black line.
I'm not sure if this lies with the unswizzle algorithm, mip0 calculation, or LZ decompression.
.\Warframe-Exporter-Advanced.exe --internal-path /Lotus/Characters/Tenno/Excalibur --cache-dir Z:\steam\232461576962714068\Cache.Windows --extract-textures --pre-ensmall-1
The Oodle compression library is added for only 1 function: OodleLZ_Decompress
. It seems this function is single-threaded, because it throws a temper tantrum when multi-threaded (see below for errors). According to the function signature, there are parameters to allocate memory for decompression operations. I assume this will allow for separated memory, thus thread-safe execution. But I have yet to implement this in LotusLib, and it would prove difficult considering the layers of abstractions I added between the user interface and actual file decompression (my actions have finally yielded consequences).
This is noticeable when attempting to preview models in the UI while trying to index vertex colors in the background (currently disabled).
Oodle complaining:
OODLE ERROR : corruption : inconsistency detected during phase 2
OODLE ERROR : LZ corruption : DecodeOneQuantum fail!
OODLE ERROR : LZ corruption : OodleLZ_Decompress failed (0 != 335)
OODLE ERROR : corruption : inconsistency detected during phase 2
OODLE ERROR : LZ corruption : DecodeOneQuantum fail!
OODLE ERROR : LZ corruption : OodleLZ_Decompress failed (0 != 1446)
OODLE ERROR : corruption : inconsistency detected during phase 2
OODLE ERROR : LZ corruption : DecodeOneQuantum fail!
OODLE ERROR : LZ corruption : OodleLZ_Decompress failed (0 != 262144)
OODLE ERROR : corruption : bad packets
OODLE ERROR : LZ corruption : DecodeOneQuantum fail!
OODLE ERROR : LZ corruption : OodleLZ_Decompress failed (0 != 219)
OODLE ERROR : corruption : inconsistency detected during phase 2
OODLE ERROR : LZ corruption : DecodeOneQuantum fail!
OODLE ERROR : LZ corruption : OodleLZ_Decompress failed (0 != 1926)
OODLE ERROR : corruption : inconsistency detected during phase 2
OODLE ERROR : LZ corruption : DecodeOneQuantum fail!
OODLE ERROR : LZ corruption : OodleLZ_Decompress failed (0 != 1671)
From PrVfRu#2715 on discord:
I'll throw a wild guess and propose that what was extracted as vertex tint before(which were deemed broken) were not JUST vertex tints but vertex wear zone +vertex material tints(or maybe something else) + vertex colors assembled into 1 attribute
here`s my proof:
1st ss: wall from ingame. wear is absent on edges, 3 colors. white, red and green
2nd ss: wall from blender, lower one displays red channel from col.001. edges are black. upper one displays blue channel from col.001 with color ramp applied to separate 0-1 range into 4 tints. same deal with floor part behind them.
2.6 alpha 3 terrain is extracted. Later versions do not extract terrains.
Mesh=/Lotus/Levels/CivilianHubs/ObjectiveRooms/ObjEidolonSectorICove/TerrainZ.tmesh
Material=/Lotus/Objects/Ostron/Natural/Terrain/EidolonPlainsRemasterTerrainNoSplat
from RAW file
The library used for glTF model exporting (fx-gltf) uses uint32_t
for buffer sizes. Models greater than 4Gb will cause this to overflow and result in a corrupted model. It's possible to fork the repository and patch this, but I attempted and ran into multiple issues.
The only trigger of this issue is Orb Vallis: /Lotus/Levels/CivilianHubs/VenusLandscape.level
Packages.bin has INI metadata for all files, including "virtual" files (i.e. mod card textures and materials, possibly solving #17.) This file used to be uncompressed (and quite large) but for a few years now the format is compressed with ZSTD (similar to #20)
At the moment this format stumps me a bit, here's what I have parsed so far (for packages.bin version 40)
struct STR {
u32 len;
char text[len];
};
struct REF {
STR name;
u32 unknown;
};
struct ENTITY {
STR package;
STR file;
u8 unknown1;
if (parent.version >= 36) {
u16 unknown2;
} else {
u32 unknown2;
}
STR inherits;
if (parent.version < 40) {
u32 unknown3; // 0
}
// READ from string_buffer until 0 byte starting at where the last entity ended.
// there is no string offset value.
// maybe this is what version 34's buffer1 array is for?
};
struct HEADER {
u8 hash[16];
u32 header_size; // 20
u32 version; // 40
u32 flags; // 1
if (version >= 40) {
u32 unknown; // ??
}
if (version >= 36) {
u32 type_count;
REF types[type_count];
}
u32 package_count; // 0 since version 34
REF packages[package_count];
if (version >= 34) {
u32 buffer1_count;
u8 buffer1[buffer1_count]; // a lot of 0xFF bytes?
u32 buffer2_count;
u8 buffer2[buffer2_count]; // suspected zstd compressed data, no header. starts with 0x100000.
u32 zdict_count;
u8 zdict[zdict_count];
} else {
u32 string_buffer_count;
u8 string_buffer[string_buffer_count];
}
u32 entity_count;
ENTITY entities[entity_count];
};
HEADER packages @ 0;
So I accidentally hit CTRL+ENTER before actually typing the issue, sorry!
The actual issue:
It would be nice to have some more info in the "Building" Section that building on Linux can be done by:
Engine/Source/Runtime/OodleDataCompression/Sdks/2.9.5/lib/
to ./bin/
.Example file list of ./bin/
for my successful build:
$ ls -1N bin/
Android
IOS
Linux
LinuxArm64
Mac
TVOS
Win32
Win64
Below are examples.
/Lotus/Characters/Tenno/JetPacks/HarrierSystem/Modular/ModularElytron/ElytronWing_skel.fbx
/Lotus/Characters/Duviri/DuriviOperaSinger/DuviriOperaSingerOutfit_skel.fbx
There are 2 possibilities to export block-compressed textures as DDS. This deals with BC1-BC7 formats.
The ddspp library used by this project currently exports all compressed DDS textures using the second option. Personally, I've found more programs are compatible with the first option, notably GIMP. Of course this is a personal observation, so I have not gathered statistics on what programs/utilities support which format.
There are 2 options to enhance compatibility with other programs:
Needing to uncheck this option is confusing for newcomers. It should not be required.
I believe this is related to the Reverse bind matrices I added at some point. I don't believe they're necessary and are just causing this issue.
While using the --debug-models
command, it ran into error LotusLib::DecomoressionException
and didn't handle it, crashing the program.
We can get all the in-game text.
This issue concerns only uncompressed textures.
The masks currently are set as 0x000000FF
0x0000FF00
0x00FF0000
for the respectively R, G and B bit masks but should be 0x00FF0000
0x0000FF00
0x000000FF
. See the texture for /Lotus/Weapons/Tenno/Zariman/LongGuns/SemiAutoRifle/ZarimanRifle_n.png
for example.
Here's an Languages.bin Parser in C# for v35. If you have time, you can convert to c++.
Dictionary<string, List<string>> ParseLanguageBin(string fileName) {
var subtitleDict = new Dictionary<string, List<string>>();
using var file = File.OpenRead(fileName);
using var reader = new BinaryReader(file);
var hash = reader.ReadBytes(16);
var version = reader.ReadInt32();
var unkA = reader.ReadInt32();
var unkB = reader.ReadInt32();
var langCodes = new List<string>();
var langCount = reader.ReadInt32();
for (var i = 0; i < langCount; i++) {
var lang = Encoding.UTF8.GetString(reader.ReadBytes(reader.ReadInt32()));
langCodes.Add(lang);
}
while (reader.BaseStream.Position != reader.BaseStream.Length) {
var contentSize = reader.ReadInt32();
var zstdDict = reader.ReadBytes(contentSize);
var unk0 = reader.ReadInt32();
using var options = new DecompressionOptions(zstdDict, new Dictionary<ZSTD_dParameter, int>() {
{ (ZSTD_dParameter)1000, 1 }
});
using var decompressor = new Decompressor(options);
for (var k = 0; k < unk0; ++k) {
var s1Len = reader.ReadInt32();
var s1 = Encoding.UTF8.GetString(reader.ReadBytes(s1Len));
// Console.WriteLine($"{reader.BaseStream.Position} {reader.BaseStream.Length} {s1}");
var s2Len = reader.ReadInt32();
var s2Bytes = reader.ReadBytes(s2Len);
var unk1 = reader.ReadInt32(); // array 1 len
for (var i = 0; i < unk1; ++i) {
var s3Len = reader.ReadInt32();
var s3Bytes = reader.ReadBytes(s3Len); // not null terminate
var s3 = Encoding.UTF8.GetString(s3Bytes);
var s2Skip = reader.ReadInt32();
var s2Take = reader.ReadUInt16();
var s2Flag = reader.ReadUInt16();
var s2Slice = s2Bytes.AsSpan(s2Skip, s2Take).ToArray();
var s2 = Encoding.UTF8.GetString(s2Slice);
if ((s2Flag & 0x200) != 0) {
using var br = new BinaryReader(new MemoryStream(s2Slice));
var dstLen = br.Read7BitEncodedInt();
var srcBuf = br.ReadBytes((int)(br.BaseStream.Length - br.BaseStream.Position));
var dst = new byte[dstLen];
decompressor.Unwrap(srcBuf, dst, false);
s2 = Encoding.UTF8.GetString(dst);
}
// Console.WriteLine($"{s1} {s3} {s2Flag:X4} {s2Take} {s2Skip} {s2Bytes.Length} {zstdDict.Length} {s2}");
subtitleDict.TryAdd(s3, []);
subtitleDict[s3].Add(s2);
}
}
}
return subtitleDict;
}
Some models use a relative position instead of absolute. For example inside the level /Lotus/Levels/CivilianHubs/ObjectiveRooms/ObjEidolonSectorICove.level
, there exists an entry with attribute Mesh=/Lotus/Objects/Ostron/Structural/OrokinRuns/OroPermieterPylon.fbx
. There is likely a hierarchy of models.
Currently this prevents objects from being placed in the scene.
Animation extraction has 3 parts: position, rotation, and scale. Currently I believe scale and rotation are fully working. Position however, is not. Reading position data has 3 parts (I believe): Read the integer value from the file, convert that integer value to the 0.0-1.0 range, then scale that range to the original scale. That could be -10.0 to 10.0 or -20.0 to 5.5, or whatever the original model contained.
I have a good idea where it's stored, but I cannot figure out the extraction method. The models /Lotus/Animations/Infested/InfSpawner/Lean*_anim.fbx
are great test subjects because 1 animation has forward, backward, left, and right versions. The Imhex script on the animation branch currently parses animations, but skips important data. Within the action
struct read inside skeletons (more specifically, between the name
and strideLen
fields, should be the position scale. I think. Probably.
More information: https://takinginitiative.wordpress.com/2020/03/07/an-idiots-guide-to-animation-compression/
May be other examples, but I cannot find them. It seems to be an issue with face indices.
/Lotus/Characters/Tenno/Operator/Heads/OperatorHeadMale_skel.fbx
/Lotus/Characters/Tenno/Operator/Heads/OperatorHeadFemale_skel.fbx
/Lotus/Characters/Tenno/Operator/Heads/AdultOperatorMale_skel.fbx
/Lotus/Characters/Tenno/Operator/Heads/AdultOperatorFemale_skel.fbx
3D models are exported with empty materials, but the materials are named and assigned to the correct vertices. The material names are exported directly with no formatting. Sometimes the material name length exceeds the Blender limit of 64 characters.
Possible solutions:
/Lotus/Characters/Tenno/Excalibur/SomeMaterial
. This can be trimmed to SomeMaterial
.Material information is useful to model extractors because it provides used textures, default colors, and many other material parameters introduced as materials have grown more complex. Currently the extractor looks for attributes in the Common Header and writes that to a text file. However newer Warframes have been missing these attributes. It's possible materials have a hierarchy for parameters, but texture information should be stored uniquely for each model. I have been unable to find where these missing material parameters are.
As a test case, /Lotus/Characters/Tenno/Dagath/DagathBodyMat
contains nothing useful. Her weapon however does contain parameters that can be used on her main model /Lotus/Weapons/Tenno/Melee/Swords/TnDagath/BladeWhip/TnDagathBladeWhipMat
.
This model is an exception in multiple areas. Once the Reverse Bind matrices are read, the file position lies at the vertices. But in the 272 reader (and seemingly true for every other 272 model), there are a few more things to be read (see below). So something major in the 272 format is missing here.
/Lotus/Characters/Tenno/Styanax/ExaltedWeapons/StyanaxDeluxe/StyanaxDeluxeExaltedSpear_skel.fbx
Cross-platform audio playback (with a simple UI with pause/play and time scrubbing) is not a simple task in Qt currently. In addition, it should support PCM and Opus codecs. There are a few audio player apps built with Qt, but no framework for building your own widget.
As I write this, I realize Qt seems to support the audio formats/codecs in this repository. But I have not confirmed this.
This is not something I plan on working on, but eventually I may want a working audio playback solution in Qt, in which case I may revisit this.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.