puxtril / warframe-exporter Goto Github PK

Lightmap also has textures, which has not been known to contain useful textures until recently. I've discovered 2 new compression formats alongside the usual BC1-7 and 1 uncompressed format.

12 /Lotus/Levels/CorpusGasCityRemaster/GasExit01/LevelCapture_mm.png
30 /Lotus/Levels/ClanDojo/ClanDojoDuviriDeadEnd/Landscape0/T0u0.lchm

Static models do not import with the correct rotation relative to their related rigged models. As an example, this is Ember with some armor attachments rotated 180 degrees on the Z axis.

Cross-platform audio playback (with a simple UI with pause/play and time scrubbing) is not a simple task in Qt currently. In addition, it should support PCM and Opus codecs. There are a few audio player apps built with Qt, but no framework for building your own widget.

As I write this, I realize Qt seems to support the audio formats/codecs in this repository. But I have not confirmed this.

This is not something I plan on working on, but eventually I may want a working audio playback solution in Qt, in which case I may revisit this.

This issue concerns only uncompressed textures.

The masks currently are set as 0x000000FF 0x0000FF00 0x00FF0000 for the respectively R, G and B bit masks but should be 0x00FF0000 0x0000FF00 0x000000FF. See the texture for /Lotus/Weapons/Tenno/Zariman/LongGuns/SemiAutoRifle/ZarimanRifle_n.png for example.

This model is an exception in multiple areas. Once the Reverse Bind matrices are read, the file position lies at the vertices. But in the 272 reader (and seemingly true for every other 272 model), there are a few more things to be read (see below). So something major in the 272 format is missing here.

1. Bone pos/rot
2. 2 unknown attributes for the BoneTree
3. Skip of 8 bytes

/Lotus/Characters/Tenno/Styanax/ExaltedWeapons/StyanaxDeluxe/StyanaxDeluxeExaltedSpear_skel.fbx

From PrVfRu#2715 on discord:

I'll throw a wild guess and propose that what was extracted as vertex tint before(which were deemed broken) were not JUST vertex tints but vertex wear zone +vertex material tints(or maybe something else) + vertex colors assembled into 1 attribute
here`s my proof:

1st ss: wall from ingame. wear is absent on edges, 3 colors. white, red and green
2nd ss: wall from blender, lower one displays red channel from col.001. edges are black. upper one displays blue channel from col.001 with color ramp applied to separate 0-1 range into 4 tints. same deal with floor part behind them.

Open-world areas have (in addition to many others) their own model format for landscapes. Inside the cache, they often have the .lndscp file extension, but they always have the Enum value 42 (as of writing this). I have not taken a close look at these, but these would be very good to support full level extraction.

Examples provided by the --print-enums flag:

/Lotus/Levels/InfestedMicroplanet/InfFleshscapeA/2.lndscp
/Lotus/Levels/InfestedMicroplanet/InfFleshscapeA/12.lndscp
/Lotus/Levels/Duviri/WarframeArenas/WFArenasEchoesOfDuviri/FractalPasturesCombat/0.lndscp
/Lotus/Levels/Duviri/MainlandAEast/0.lndscp
/Lotus/Levels/TheNewWar2021/Part1/CetusFire/0.lndscp
/Lotus/Levels/TheNewWar2021/VenusLandscapePostWar/0.lndscp
/Lotus/Levels/TheNewWar2021/CetusPostWar/0.lndscp
/Lotus/Levels/TheNewWar2021/PlainsLandscapePostWar/0.lndscp
/Lotus/Levels/NightWave/TheGlassMaker/CetusCrimeScene/0.lndscp
/Lotus/Levels/CivilianHubs/CetusRemaster/0.lndscp

The library used for glTF model exporting (fx-gltf) uses uint32_t for buffer sizes. Models greater than 4Gb will cause this to overflow and result in a corrupted model. It's possible to fork the repository and patch this, but I attempted and ran into multiple issues.

The only trigger of this issue is Orb Vallis: /Lotus/Levels/CivilianHubs/VenusLandscape.level

Here's an Languages.bin Parser in C# for v35. If you have time, you can convert to c++.

Dictionary<string, List<string>> ParseLanguageBin(string fileName) {
    var subtitleDict = new Dictionary<string, List<string>>();

    using var file = File.OpenRead(fileName);
    using var reader = new BinaryReader(file);
    
    var hash = reader.ReadBytes(16);
    var version = reader.ReadInt32();
    var unkA = reader.ReadInt32();
    var unkB = reader.ReadInt32();

    var langCodes = new List<string>();
    var langCount = reader.ReadInt32();
    for (var i = 0; i < langCount; i++) {
        var lang = Encoding.UTF8.GetString(reader.ReadBytes(reader.ReadInt32()));
        langCodes.Add(lang);
    }

    while (reader.BaseStream.Position != reader.BaseStream.Length) {
        var contentSize = reader.ReadInt32();
        var zstdDict = reader.ReadBytes(contentSize);

        var unk0 = reader.ReadInt32();

        using var options = new DecompressionOptions(zstdDict, new Dictionary<ZSTD_dParameter, int>() {
            { (ZSTD_dParameter)1000, 1 }
        });
        using var decompressor = new Decompressor(options);

        for (var k = 0; k < unk0; ++k) {
            var s1Len = reader.ReadInt32();
            var s1 = Encoding.UTF8.GetString(reader.ReadBytes(s1Len));

            // Console.WriteLine($"{reader.BaseStream.Position} {reader.BaseStream.Length} {s1}");

            var s2Len = reader.ReadInt32();
            var s2Bytes = reader.ReadBytes(s2Len);

            var unk1 = reader.ReadInt32(); // array 1 len

            for (var i = 0; i < unk1; ++i) {
                var s3Len = reader.ReadInt32();
                var s3Bytes = reader.ReadBytes(s3Len); // not null terminate
                var s3 = Encoding.UTF8.GetString(s3Bytes);

                var s2Skip = reader.ReadInt32();
                var s2Take = reader.ReadUInt16();
                var s2Flag = reader.ReadUInt16();

                var s2Slice = s2Bytes.AsSpan(s2Skip, s2Take).ToArray();
                var s2 = Encoding.UTF8.GetString(s2Slice);

                if ((s2Flag & 0x200) != 0) {
                    using var br = new BinaryReader(new MemoryStream(s2Slice));
                    var dstLen = br.Read7BitEncodedInt();
                    var srcBuf = br.ReadBytes((int)(br.BaseStream.Length - br.BaseStream.Position));
                    var dst = new byte[dstLen];
                    decompressor.Unwrap(srcBuf, dst, false);
                    s2 = Encoding.UTF8.GetString(dst);
                }

                // Console.WriteLine($"{s1} {s3} {s2Flag:X4} {s2Take} {s2Skip} {s2Bytes.Length} {zstdDict.Length} {s2}");
                subtitleDict.TryAdd(s3, []);
                subtitleDict[s3].Add(s2);
            }
        }
    }

    return subtitleDict;
}

Hopefully in the near future until the animation is exported

We can get all the in-game text.

Reported by a user on Discord. Could not get this duplicated on my system, will need to investigate more.

Animation extraction has 3 parts: position, rotation, and scale. Currently I believe scale and rotation are fully working. Position however, is not. Reading position data has 3 parts (I believe): Read the integer value from the file, convert that integer value to the 0.0-1.0 range, then scale that range to the original scale. That could be -10.0 to 10.0 or -20.0 to 5.5, or whatever the original model contained.

I have a good idea where it's stored, but I cannot figure out the extraction method. The models /Lotus/Animations/Infested/InfSpawner/Lean*_anim.fbx are great test subjects because 1 animation has forward, backward, left, and right versions. The Imhex script on the animation branch currently parses animations, but skips important data. Within the action struct read inside skeletons (more specifically, between the name and strideLen fields, should be the position scale. I think. Probably.

More information: https://takinginitiative.wordpress.com/2020/03/07/an-idiots-guide-to-animation-compression/

Level files has override function where material of original mesh can be switched with another one. This behaviour is broken for several extractor updates

Some models use a relative position instead of absolute. For example inside the level /Lotus/Levels/CivilianHubs/ObjectiveRooms/ObjEidolonSectorICove.level, there exists an entry with attribute Mesh=/Lotus/Objects/Ostron/Structural/OrokinRuns/OroPermieterPylon.fbx. There is likely a hierarchy of models.

Currently this prevents objects from being placed in the scene.

Packages.bin has INI metadata for all files, including "virtual" files (i.e. mod card textures and materials, possibly solving #17.) This file used to be uncompressed (and quite large) but for a few years now the format is compressed with ZSTD (similar to #20)

At the moment this format stumps me a bit, here's what I have parsed so far (for packages.bin version 40)

struct STR {
    u32 len;
    char text[len];
};

struct REF {
    STR name;
    u32 unknown;
};

struct ENTITY {
    STR package;
    STR file;
    u8 unknown1;
    if (parent.version >= 36) {
        u16 unknown2;
    } else {
        u32 unknown2;
    }
    STR inherits;
    if (parent.version < 40) {
        u32 unknown3; // 0
    }
    
    // READ from string_buffer until 0 byte starting at where the last entity ended.
    // there is no string offset value.
    // maybe this is what version 34's buffer1 array is for?
};

struct HEADER {
    u8 hash[16];
    u32 header_size; // 20
    u32 version; // 40
    u32 flags; // 1
    if (version >= 40) {
        u32 unknown; // ??
    }

    if (version >= 36) {
        u32 type_count;
        REF types[type_count];
    }

    u32 package_count; // 0 since version 34
    REF packages[package_count];

    if (version >= 34) {
        u32 buffer1_count;
        u8 buffer1[buffer1_count]; // a lot of 0xFF bytes?
        u32 buffer2_count;
        u8 buffer2[buffer2_count]; // suspected zstd compressed data, no header. starts with 0x100000.
        u32 zdict_count;
        u8 zdict[zdict_count];
    } else {
        u32 string_buffer_count;
        u8 string_buffer[string_buffer_count];
    }

    u32 entity_count;
    ENTITY entities[entity_count];
};

HEADER packages @ 0;

v2.5.1 errors with This is a console program. It needs to be executed inside a terminal (like Powershell). despite being run via WSL. Example screenshot is below.

3D models are exported with empty materials, but the materials are named and assigned to the correct vertices. The material names are exported directly with no formatting. Sometimes the material name length exceeds the Blender limit of 64 characters.

Possible solutions:

• Clamp material lengths to 64 characters (not good)
• Remote the beginning material path length. Often what causes these to exceed 64 characters is a full path: ex. /Lotus/Characters/Tenno/Excalibur/SomeMaterial. This can be trimmed to SomeMaterial.
• Implement the above, but also export the full path to Blender's Custom Properties section. I'm not sure if this is possible with glTF.

2.6 alpha 3 terrain is extracted. Later versions do not extract terrains.

Mesh=/Lotus/Levels/CivilianHubs/ObjectiveRooms/ObjEidolonSectorICove/TerrainZ.tmesh
Material=/Lotus/Objects/Ostron/Natural/Terrain/EidolonPlainsRemasterTerrainNoSplat
from RAW file

So I accidentally hit CTRL+ENTER before actually typing the issue, sorry!

The actual issue:
It would be nice to have some more info in the "Building" Section that building on Linux can be done by:

• Downloading Unreal Engine from the official Linux page (though logging in is required)
• Extracting the zip's Engine/Source/Runtime/OodleDataCompression/Sdks/2.9.5/lib/ to ./bin/.

Example file list of ./bin/ for my successful build:

$ ls -1N bin/
Android
IOS
Linux
LinuxArm64
Mac
TVOS
Win32
Win64

I believe this is a dynamic system to place models on levels. They are extracting but are placed in the incorrect (up) position. /Lotus/CivilianHubs/ObjectiveRooms/ObjEidolonSectorICove.level

Material information is useful to model extractors because it provides used textures, default colors, and many other material parameters introduced as materials have grown more complex. Currently the extractor looks for attributes in the Common Header and writes that to a text file. However newer Warframes have been missing these attributes. It's possible materials have a hierarchy for parameters, but texture information should be stored uniquely for each model. I have been unable to find where these missing material parameters are.

As a test case, /Lotus/Characters/Tenno/Dagath/DagathBodyMat contains nothing useful. Her weapon however does contain parameters that can be used on her main model /Lotus/Weapons/Tenno/Melee/Swords/TnDagath/BladeWhip/TnDagathBladeWhipMat.

While using the --debug-models command, it ran into error LotusLib::DecomoressionException and didn't handle it, crashing the program.

Below are examples.

/Lotus/Characters/Tenno/JetPacks/HarrierSystem/Modular/ModularElytron/ElytronWing_skel.fbx
/Lotus/Characters/Duviri/DuriviOperaSinger/DuviriOperaSingerOutfit_skel.fbx

Needing to uncheck this option is confusing for newcomers. It should not be required.

I believe this is related to the Reverse bind matrices I added at some point. I don't believe they're necessary and are just causing this issue.

The Oodle compression library is added for only 1 function: OodleLZ_Decompress. It seems this function is single-threaded, because it throws a temper tantrum when multi-threaded (see below for errors). According to the function signature, there are parameters to allocate memory for decompression operations. I assume this will allow for separated memory, thus thread-safe execution. But I have yet to implement this in LotusLib, and it would prove difficult considering the layers of abstractions I added between the user interface and actual file decompression (my actions have finally yielded consequences).

This is noticeable when attempting to preview models in the UI while trying to index vertex colors in the background (currently disabled).

Oodle complaining:

OODLE ERROR : corruption : inconsistency detected during phase 2
OODLE ERROR : LZ corruption : DecodeOneQuantum fail!
OODLE ERROR : LZ corruption : OodleLZ_Decompress failed (0 != 335)
OODLE ERROR : corruption : inconsistency detected during phase 2
OODLE ERROR : LZ corruption : DecodeOneQuantum fail!
OODLE ERROR : LZ corruption : OodleLZ_Decompress failed (0 != 1446)
OODLE ERROR : corruption : inconsistency detected during phase 2
OODLE ERROR : LZ corruption : DecodeOneQuantum fail!
OODLE ERROR : LZ corruption : OodleLZ_Decompress failed (0 != 262144)
OODLE ERROR : corruption : bad packets
OODLE ERROR : LZ corruption : DecodeOneQuantum fail!
OODLE ERROR : LZ corruption : OodleLZ_Decompress failed (0 != 219)
OODLE ERROR : corruption : inconsistency detected during phase 2
OODLE ERROR : LZ corruption : DecodeOneQuantum fail!
OODLE ERROR : LZ corruption : OodleLZ_Decompress failed (0 != 1926)
OODLE ERROR : corruption : inconsistency detected during phase 2
OODLE ERROR : LZ corruption : DecodeOneQuantum fail!
OODLE ERROR : LZ corruption : OodleLZ_Decompress failed (0 != 1671)

Currently, the texture resolution needs to be calculated dynamically. Warframe provides a resolution, but this should only be used for the texture resolution ratio. The formula to calculate the actual resolution is a bit clunky.

Thanks to a discovery by @sehnryr (detailed in sehnryr/wfcache-api#1), this will be a lot simpler. Not a huge priority as the current implementation is fast enough and works without issue.

There are 2 possibilities to export block-compressed textures as DDS. This deals with BC1-BC7 formats.

1. Do not include a DXT10 header, place a value in the FOURCC field of the DDS Header.
2. Include a DXT10 header and place a value in the DXGI_FORMAT field, leave the FOURCC value as DXT10.

The ddspp library used by this project currently exports all compressed DDS textures using the second option. Personally, I've found more programs are compatible with the first option, notably GIMP. Of course this is a personal observation, so I have not gathered statistics on what programs/utilities support which format.

There are 2 options to enhance compatibility with other programs:

1. Fork the ddspp library and modify behavior to match previously mentioned option 1.
2. Include another library to export textures as PNG rather than DDS. This would add maximum compatibility with other programs, but would definitely require a separate library, and logic to convert DDS to PNG. I choose PNG because it supports transparency, and is the format used by Warframe developers on the backend, as indicated by file extensions within the archive.

Textures extracted pre-ensmallening have a different compression in the cache files and are processed differently as textures. In the cache, they use LZ decompression. As textures, they need to be unswizzled before written to disk. Textures pre-ensmallening are missing data at the bottom - There's a black line.

I'm not sure if this lies with the unswizzle algorithm, mip0 calculation, or LZ decompression.

.\Warframe-Exporter-Advanced.exe --internal-path /Lotus/Characters/Tenno/Excalibur --cache-dir Z:\steam\232461576962714068\Cache.Windows --extract-textures --pre-ensmall-1

May be other examples, but I cannot find them. It seems to be an issue with face indices.

/Lotus/Characters/Tenno/Operator/Heads/OperatorHeadMale_skel.fbx
/Lotus/Characters/Tenno/Operator/Heads/OperatorHeadFemale_skel.fbx
/Lotus/Characters/Tenno/Operator/Heads/AdultOperatorMale_skel.fbx
/Lotus/Characters/Tenno/Operator/Heads/AdultOperatorFemale_skel.fbx