laupetin / openassettools Goto Github PK

Happens with this fastfile

plutonium_common_mp.zip

Using the latest OAT on this fastfile there is an issue with the localized assets.

The steps to reproduce should be the following:

1. Unlink the mod.ff
2. Take the zone_raw and try to relink it with the Linker.

If the issue is reproducible it should say it is unable to find localized asset for "mod".

Currently you have to manually specify the fastfile you want to unlink and only that fastfile will be dumped. However, I'm suggesting a command that just loads every fastfile in the search path sequentially and then dumps each fastfile to disk like the normal unlink functionality. This way a modder could quickly generate a dump of all the assets in a game, and then they'd have a similar level of modding access to raw assets like the official mod tools provide for other CoD games.

The field WeaponDef::parentWeaponName should be a reference to a weapon asset. This applies to both T5 and T6.

Without setting the IMAGE_FILE_LARGE_ADDRESS_AWARE flag a 32bit application will only be able to allocate as much as 2GB memory (see also).

Since both Unlinker and Linker can load any amount of fastfiles beforehand, setting this option will increase the amount of possible loaded fastfiles by a lot.
It's not really an urgent issue but there should not be a downside to it really.
With premake this is probably done by using this: https://premake.github.io/docs/largeaddressaware/

confirmed crash on mp_radar (Outpost).

it seems Downturn will also course troubles.

After the introduction of marking in 0.6.0 there are assets that lead to an infinite recursion when trying to mark them. This can be due to a struct being referenced by pointer a nested struct.
This happens for example with menus and the support structure in Statements.

The bug leads to unlinker and linker just exiting early without a log message due to a stack overflow.

Currently sounds cannot be added or modified using OAT because the game uses a special storage structure for sounds. Streamed sounds are stored in .sabs extension files and loaded sounds are stored in .sabl extension files. In order to allow for making mods that have custom or modified sounds the sabs/sabl structures need to be reversed. Additionally the sound/snddriverglobals/sndpatch assets need to be dumpable and linkable to allow modifying the sound aliases which the engine uses to know how to play the sound.

So what needs to be done is the following:

• sabs/sabl dumping
• sabs/sabl writing
• sound/snddriverglobals/sndpatch asset dumping
• sound/snddriverglobals/sndpatch asset writing

https://github.com/master131/Black-Ops-II-Sound-Studio This repo has useful research about the sabs/sabl structures and can already export sounds from a sabs/sabl file.

I relinked every single menu on IW5 and although I might have missed a few I only found one error.

The gameplay options menu is missing one background box for the game play options

The menu in question is settings_recipe_rules_base.menu (not 100% sure)

and can be found here https://github.com/diamante0018/iw5-menus/blob/main/zone_raw/localized_ui_mp/ui_mp/settings_recipe_rules_base.menu

C++20 added two new directives to the preprocessor: #elifdef and #elifndef

With this code I added them to IW4x lPrecomp.cpp

Would be cool if OAT can also provide these new directives

In CoDs older than T6 IWDs were used to store assets for raw loading such as rawfiles, scripts, images, streamed sounds etc. An IWD is just a normal zip file compressed using zlib, just with .iwd extension. While not strictly necessary to add since it is trivial to just zip up the files and rename the file extension to iwd, the canonical mod tools do have an iwd list for the linker to know what files go into iwds and then the linker will create the iwd with the assets.

Linker should display an error message when the name of the game in a zone file is unrecognized.
Currently there is no error message at all.

As encountered in pc_options_gamepad.menu from IW4x:

PC_OPTIONS_STRLIST_RAW(3, "@MENU_BUTTON_LAYOUT", "gpad_buttonConfig", 
	{"@MENU_DEFAULT"; "buttons_default";
	"@MENU_TACTICAL"; "buttons_tactical";
	"@MENU_LEFTY"; "buttons_lefty";
	"@MENU_NOMAD"; "buttons_nomad";
	"@MENU_DEFAULT_ALT"; "buttons_default_alt";
	"@MENU_TACTICAL_ALT"; "buttons_tactical_alt";
	"@MENU_LEFTY_ALT"; "buttons_lefty_alt";
	"@MENU_NOMAD_ALT"; "buttons_nomad_alt"}, execNow "bindgpbuttonsconfigs";, ;, ON_GPAD_DISABLED, ON_GPAD_ENABLED)

does not correctly expand.

When everything is placed in the same line, the preprocessor can parse the macro usage.

Currently the only way to interact with Linker and Unlinker is via command line arguments.
There are some scripts in the community that do some sorts of text-based user interface to make controlling the tool easier.

Having a TUI integrated would allow for easier usage and allows more capabilities than running an external script.

OAT can dump some stock menus but the game cannot recompile them.

Here is a list of three menus and a summary of the errors.

https://pastebin.com/wRnwHHf8 (youarehost)
line 89 exp rect x (- 854) + ((float(milliseconds() % 50000) / 50000) * (854));
line 99 ^^
line 161 exp rect y (0 - 0) + ((24 + 3 * 20));
line 172 ^^
line 183 ^^

https://pastebin.com/T3ufDEN8 (player_popup_privatematch)

ERROR: redefinition of expression for 'disabled'.
Menu load error: ui_mp\player_popup_privatematch.menu, line 242: error parsing the itemdef keyword exp's parameters
Menu load error: ui_mp\player_popup_privatematch.menu, line 242: couldn't parse menu keyword itemDef

The error above looks like the devs made a mistake and it's not OAT's fault.

https://pastebin.com/JX45Vs6V (setting bonus map menu)

Menu load error: ui_mp\settings_bonus_map.menu, line 39: unknown itemdef keyword

This combination is currently redundant and done almost everywhere.
It can be simplified to a calloc call.
Since this uses the MemoryManager class, this method has to first be introduced.

Making it a template with a typename argument would also simplify the usage instead of having to always specify the size and then cast.

Example of how it could look like:

auto* material = memory.Calloc<Material>();
auto* fxArray = memory.Calloc<FxEffectDef>(5);

The weapon asset knife_zm_alcatraz from zm_prison does reference the anim viewmodel_M4m203_knife_melee_1 (as its meleeAnim) with a name that is non-lowercase.

The actual asset however is called viewmodel_m4m203_knife_melee_1.
I suppose all assets have a lowercase name then?

Currently there is no documentation about how to create a new project other than unlinking an existing zone and using its output as a base.

Currently, everytime the linker runs it builds every asset entirely new from its source material.
This can include parsing source files, processing them and converting the data to the target game.

This process can be shorted for some assets by caching the resulting asset and only processing the input again, if itself or one of its included files was changed (e.g. for menus).

This process however is only worth doing on assets that take a considerable amount of effort to build.
For example caching something like stringtables would probably not grant any noticable performance increase.

Depending on how the assets are parsed this cannot happen on an "per asset" basis though.
For example menus must be cached on a per file basis, since one file can include a menulist or 0,1,* menus.

For file based sources the caching can check for changes by just considering the filesystem's last modified timestamp (This is also how MSVC seems to do it).
For gdt based sources it will probably be necessary to use a checksum like CRC.

It is important to consider all source files for an asset instead of just the entry file.
For example a menu can be located inside file X which included file Y however, which defines a macro.
In this case, the cache must be considered dirty, even when only file Y changes.

Now that IPak writing is implement there should be documentation about how IPaks are structured since it is rather complicated and it took a bit of time to reinvestigate how they worked.

Most of the terms being used in the code are also made up since there is no official information about how its parts are called.

I changed the playerAnimType name for snipers to allow specialty_fastads to work with them, but I can't build with the custom playerAnimType.

I replaced the playerAnimType name in mp/playeranimtypes.txt and zm/playeranim.script also.

I was trying to add Tac-45 Dual Wield to T6ZM but could not get the anims to work properly no matter what I tried.

Example project:
zm_fnp45.zip

The following menu file was chosen because it is relatively small and cannot be used by the game.
I tried both formats with or without function definitions.
I will include the menu without function definitions.

patch_mp_menus.txt menulist can be used to override any menu in the game.

https://pastebin.com/YLixTKqE

The menu was not modified in any way. taken as it is from the dumper

In this case, my CI thinks everything is fine because Lineker returned 0 even though it could not do anything.

To be able to compile custom materials there needs to be a reimplementation of the original material compilation pipeline.
This includes (terms being used might be inaccurate due to missing knowledge from my side):

• Loading material properties from GDT
• Creating technical material data using material templates
• Compiling chosen techset and techniques using techset templates
• Compiling shaders for techniques using shader templates
• Applying statemaps on material statebits per technique

Since it is not possible to exactly find out how the pipeline of IW4 might have looked like, all that is possible is guessing by referencing IW3 modtools and extending functionality based on the compiled output for IW4 that is known.
Since there is for example no way to be sure about which statemaps existed or what they looked like, it will only be possible to aim for a best effort implementation that gets as close as possible to the vanilla data.

There's a difference in how the game's native parser and OAT interpret the following menu code:

{
    menuDef
    {
        name ;
    }
}

The game would interpret this value the same way as would supplying an empty string: name ""

OAT instead interprets the semicolon as the name value like this would: name ";"

There are two muzzle flash fx assets for a weapon; first person, and third person, and currently they are not automatically copied over as a dependency which causes the game to fill in the missing asset with a random asset.

If you link bowie_knife_zm from zm_buried.ff, it works fine. However, if you take that same weapon file from the unlinker and add it to your mod folder with no changes made to the weapon file and link with that weapon file, the viewmodel for that weapon does not show.

I also noticed a few other weapons were not showing the weapon name when trying to override them on a Plutonium dedi if they were also linked from your mod even though I didn't see any noticeable issues with the weapons, so they are probably also not linking correctly: insas_mp, vector_mp, sig556_mp

Is there a current or upcoming capability for the Linker to produce signed fastfiles (TAff0100) instead of unsigned ones (TAffu100) for T6?

The name of the zone definition file should not necessarly be the name of the zone.
In some cases the name of the zone must be very specific without being very descriptive.

For example for mods, the name must be mod.ff.
In case of overwriting vanilla fastfiles, the name needs to match the original fastfile.

In all games other than T6 it is at least possible to just rename the fastfile afterwards, so no big deal.
However in T6 you cannot rename a zone without breaking the ability to decrypt them again.

To prevent having to name all of your projects "mod" or the same as a vanilla fastfile name it should be possible to independently name the zone definition and the resulting fastfile

Example usecase:

Fastfile + IPak building process.

I want to be able to just specify a single project to build to first build the fastfile and afterwards an appropriate IPak for it.

When I use OAT in a workflow over at git.alterware we have a small and not-so-powerful machine doing the ci work.

so what ends up happening is when menu parsing is done it might stay "unresponsive" for several minutes because, in reality, it's writing the zone itself.

OAT prints a message when the zone is done creating, but if we are in verbose mode I'd like a message showing that zone creation has begun so I know it's not stuck parsing some menu file.

Example:
Expected output: sound\mus\zombie\tomb\egg\archangel.SN45.pc.snd.flac
Actual output: raw\sound\mus\zombie\tomb\egg\archangel.SN45.pc.snd

Currently does not show an error message.

There is currently no implementation for creating ipaks with image data for T6.

I did already write an implementation for this somewhen, the code needs to be ported to OAT.

Currently OAT does compile assets and link them in the same step.
For usability this is not a big problem since it reduces compilation to a single step.
However this prevents implementing linking of assets without having code for decompiling them to their source forms like for materials or models.

Since the materials can be quite complex and there's no guarantee for 100% accurate output and i assume this will be the same for other assets as well.
Therefore it will be useful to have a compiled form for these assets which can then be fed into a decompiler optionally or be linked again without decompilation.

Another advantage is that compiling won't have to happen on every linking since the compiled form of the assets can be cached to be reused in case the source did not change.

In the future instead of having two main tools: Linker/Unlinker, there will be four main tools:

• Linker: Links compiled assets into fastfiles/other respective formats
• Unlinker: Unlinks compiled assets from fastfiles/other respective formats
• Compiler: Takes in compiled assets and tries to recreate their original source format (best effort)
• Decompiler: Takes in assets in source form and compiles them

For example in terms of materials this will mean:

• Source form is material properties in GDT + Material compilation pipeline
• Compiled form is binary (basically unmodifiable) blob in materials/<name> raw folder

Currently IW5 menus can only be compiled by specifying --menu-permissive which does not try to validate event handler code.
To bring IW5 on the same level as IW4 menus, the available IW5 functions must be added.

Currently when using indirect asset reference it can happen from time to time that assets are not entirely lowercase.
This makes it so assets cannot be found, e.g. viewmodel_USPMGS_ADS_fire does not match the actual asset name viewmodel_uspmgs_ads_fire.

Therefore it is required to normalize the asset names the following way:

• Assets have all lowercase names
• Assets use / instead of \ as path separator (see #82 )

If you export a sound that is already in game using BO2 Sound Studio and use the same sound alias line that is given from Unlinker, the sound comes out lower than normal in game. This is especially noticeable with voicelines.

This could be an issue with exporting from BO2 Sound Studio, I'm not sure.

It's needed to compile IW4 Menus, it's makes it almost impossible without

example: name "cac_weapon_" ## weaponArg

When trying to build IW4x's menus with OAT instead of building them with the game's menu compiler, some menus seem to be off.

Examples

Serverlist

The serverlist cannot be used properly and the text on the "source" button is missing

Options

The sidebar on the left seems to have a weird white glow.

Assetlists are already usable for ignoring assets inside project definitions as well as loading all assets from an assetlist.

This is particularly useful when you want to build an IPak with all images from a built zone for example.

To enable this workflow Linker should automatically generate assetlists after successfully linking.
These assetlists should be placed in an appropriate source directory in a "assetlist" subfolder.

Would probably also good if creating assetlists could be turned off via cli args.
It should be enabled by default though for now.

I will do my best to summarize previous discord discussions so that it's documented here.

Menus included in a menulist do not have definite file paths. This can cause the unlinker to dump menus that do not exist as a separate file or dump them into the wrong folder. As a result the menus cannot be relinked.

mod.all.aliases.csv:

name,file,template,loadspec,secondary,group,vol_min,vol_max,team_vol_mod,dist_min,dist_max,dist_reverb_max,volume_falloff_curve,reverb_falloff_curve,volume_min_falloff_curve,reverb_min_falloff_curve,limit_count,limit_type,entity_limit_count,entity_limit_type,pitch_min,pitch_max,team_pitch_mod,min_priority,max_priority,min_priority_threshold,max_priority_threshold,spatialized,type,loop,randomize_type,probability,start_delay,reverb_send,duck,duck_group,pan,center_send,envelop_min,envelop_max,envelop_percentage,occlusion_level,occlusion_wet_dry,is_big,distance_lpf,move_type,move_time,real_delay,subtitle,mature,doppler,futz,context_type,context_value,compression,timescale,music,fade_in,fade_out,pc_format,pause,stop_on_death,bus,snapshot
mus_zmb_secret_song,sound\mus\zombie\tomb\egg\SleepTokenSN45.pc.snd.flac,,,,grp_music,23252,23252,,0,5000,5000,default,default,allon,allon,8,priority,8,oldest,32767,32767,,100,100,63,-1,,streamed,nonlooping,volume,-1,0,0,,snp_movie,2d,0,0,0,0,63,,no,yes,none,0,,,,no,0,0,0,,yes,yes,1,1,,yes,yes,bus_reference,

.zone file:

>game,t6
>name,mod
soundbank,mod

the rest is copied from zm_tomb

Hi, i get an issue. When i click "generate.bat" file, it appears that notification. How can i fix it?

On Modern Warfare 3 (2011) there is a fast file called localized_ui_mp

When dumped the "menu list" asset ui_mp/menus.txt will contain various function definitions.

Some weeks ago, during the relinking process of the fast file, I encountered an error I can no longer reproduce.

Error: ui_mp/menus.txt L80:38 Unknown function
FUNC_12() * FUNC_7()) + (FUNC_9() * FUNC_6());
Parsing menu file failed!
Failed to load asset of type "menulist": "ui_mp/menus.txt"

Because I forgot to document this issue when I first got it I have to trace back and provide as much info as possible.

I think the best way is to add a unit test to confirm without a doubt OAT can parse the following.

functionDef
{
        name                        "FUNC_6"
        value                       team("name") == "TEAM_ALLIES";
}

functionDef
{
        name                        "FUNC_7"
        value                       team("name") == "TEAM_AXIS";
}

functionDef
{
        name                        "FUNC_9"
        value                       tablelookup("mp/factionTable.csv", 0, getmapcustom("allieschar"), 14);
}
 
functionDef
{
        name                        "FUNC_12"
        value                       tablelookup("mp/factionTable.csv", 0, getmapcustom("axischar"), 14);
}

functionDef
{
        name                        "FUNC_15"
        value                       (FUNC_12() * FUNC_7()) + (FUNC_9() * FUNC_6());
}

The above definitions can be found in the menu list "ui_mp/menus.txt file when the fast file is dumped.

I think having a unit test for this will confirm whether the original error message I got is legit or not.

Linker invocation command I used today (no error): .\bin\Linker.exe -v --menu-permissive --menu-no-optimization -l "<game folder>\zone\english\localized_ui_mp.ff" localized_ui_mp

I could have used some other command line arguments last time triggering the error. A unit test should try to test a couple of different settings to make sure complex function definitions THAT RELY ON PREVIOUS FUNCTION DEFINITIONS are parsed correctly.

If OAT encounters a duplicated localized string it should issue a warning to make sure the user is aware that one version of a localized string is overwritten by another occurrence of it.

Otherwise there may be confusion when one occurrence of the string is changed by no change happens in the final output.

After the separation of concerns of loading and marking assets the generated code of both now may contain unused or empty methods.
To clean them up it must be ensured that:

• The loader only considers structs that are not leafs
• Scriptstrings no longer make a struct not be considered a leaf
• The marker only considers structs that are required for marking

Currently if you add an asset to your zone you can use back slashes in the name. While this works it is non standard and creates ambiguity on how to reference the asset. For example if you add a string table to your zone like this and you want to override the original mp/statstable.csv:

stringtable,mp\statstable.csv

instead of overriding the asset it will create a new asset which makes it unclear which one will actually be used since forward slashes and back slashes work the same way in paths but in cases where the name is directly compared it will be different.