underrout / lunarhelper Goto Github PK

Just an idea, but it should theoretically be possible to check Lunar Magic's registry values for a potentially suitable emulator setup for the user if they have not specified an explicit emulator via emulator_path and emulator_options.

Currently you can't have something like

patches
[
--	patches/hex_edits.asm
]

because Lunar Helper does not account for comments inside lists as far as I can tell.

For example to disable "255 sprites per level" support in pixi.

Currently the build process terminates as soon as any error is encountered. Perhaps it would make sense to simply move on to the next step instead so that more errors can be discovered within a single build.

On the other hand, if the rest of the build takes longer than fixing the error, it could be annoying to have to wait for the build to finish in order to start the new one including the fix. Still worth considering maybe.

Not all cases have easily digestible error messages yet, making it hard to tell what exactly has gone wrong if, for example, some path is left unspecified.

Currently in 4.0.0 Lunar Monitor will only be injected when Lunar Magic is launched through Lunar Monitor Loader, but it would be possible to allow both loading through Lunar Monitor Loader as well as the old method of Lunar Magic injecting itself through usertoolbar.txt. The method would be something like this:

• pass an additional command-line argument to Lunar Monitor Loader to let it know that it should inject into a running Lunar Magic
• pass this argument along from Lunar Monitor Loader to Lunar Monitor through a pipe, so that Lunar Monitor knows to initialize differently

Much of Quick Build's analysis could be streamlined via a custom built asar DLL that reports every file it assembles.

Lunar Helper could replace PIXI's, GPS's and UberASM Tool's asar DLLs with such an altered DLL and then for each tool record the source files that were actually assembled. This would make arbitrary and missing dependencies obsolete for these tools, require less overhead for analyzing dependencies and make a lot of complicated parsing unnecessary.

For AddMusicK, the situation is slightly different in that it uses an EXE of asar rather than a DLL. In addition, it is an old version (1.33b), which I can't find source code for at the moment. The oldest version I found on SMWC is 1.35 which seems to work fine with AddMusicK, but I cannot for the life of me get the source code to compile, so idk if that's gonna be viable. AMK 1.0.9 is going to use an EXE of asar 1.81 from what I've seen so maybe I'll just wait for that to become standard and then look into it more.

Open questions:

• warn users before Lunar Helper overwrites other tools DLLs? (probably, sounds reasonable)
• remove old dependency resolving method completely or keep as fallback? (probably keep it as fallback in case people wanna stick with other DLLs, someone could be using experimental new asar DLLs and not want to use an altered 1.81 DLL for example, we'll still need the old resolvers for AMK as well probably)
• best method for filtering out assembled files that were generated by the tool itself? (probably just make a list or alternatively don't filter them out at all and just accept that changing them will lead to unnecessary re-insertions, but who keeps and alters temp files anyway)
• future compatibility? (PIXI will start using 64-bit DLLs starting with version 1.40, right now all the tools use 32-bit DLLs)
• how to handle canreadfile? (File may be present or absent, maybe just use null as "hash" value for absent files so that adding the file will change the hash)

From a quick test module that defines a struct (I tend to use structs alot for organizing data arrangements, which have been frustrating from other warnings/errors I get in other tools from how asar treats them, so i acknowledge that this may be impossible of infeasible to address simply from how asar implements struct definitions), lunar helper causes an error by trying to define struct fields to a vaule which makes asar report there is an invalid label (because of the struct.field format) and throw an error. I have not looked into how asar implements structs in detail, and do not know if there is a way to disambiguate a genuine struct label and an actual improperly written label with a period in it, but I would consider it to make sense to treat them as more a define rather than a label in processing them if it is possible to disambiguate them.

Could cut down on time spent analyzing previous build result if that turns out to take too long in the future (i.e. once dependency graph is added).

Just so there's a record of what's going on and people can potentially have an easier time asking for help if something went wrong.

Apparently Lunar Magic doesn't make it so map16 pages 0x40+ are usable unless you have at least one non-empty tile on at least one of the pages between 0x40 and 0x80. Adding a tile there and deleting it after saving map16 does not work so I guess I will have to include a random altered tile on one of those pages in future initial bps patches (exciting).

Right now there's retroarch_path and retroarch_core, but the second is really just passed as an option to the exe specified by the first.
The full command is currently $"-L \"{Config.RetroArchCore}\" \"{fullRom}\"".

I think I will probably just replace these with a generic emulator_path and emulator_options variables, if that is a valid solution. It should more clearly communicate that you can set up any emulator this way and also give people more control if they ever want to use other command line flags with their emulator of choice.

This does mean that the emulator_options config var in the example config file should probably be something like

-- uncomment if you're using retroarch and replace the path with the path to your retroarch core
-- you can also use this to pass other command line parameters to your emulator of choice
-- emulator_options = -L C:\Users\user\AppData\Roaming\RetroArch\cores\snes9x_libretro.dll

hopefully that would be sufficiently clear.

Note that removing the two retroarch config variables is technically a breaking change, so I will probably make this change for v2.0.0, which will introduce the dependency graph for Quick Build, which will involve the breaking change of getting rid of the shared_folder define, which is a breaking change as well.

Could be useful to apply a bunch of patches in a specific folder without having to specify each in the patches list.

Only issue: what about patches which consist of/use multiple asm files?

Answer 1: Probably just assume that every asm file in the patches folder is a standalone patch and let users handle it either by putting extra asm files in other folders and incsrcing them or by manually specifying the file to patch in the patches list instead.

Answer 2: Wait for #14 to happen and then only insert the files from the patches folder that are sinks in the dependency graph.

Calls to these asar functions are not currently handled by Lunar Helper, i.e. changes to files accessed via these two functions will not be tracked accurately by Quick Build.

Currently UberASMTool does not set the exit code != 0 when it terminates with an error, so this is really more of a bug with UberASMTool, so just gonna wait until that gets added I guess.

Temporary ROMs stay behind in the filesystem if an error is encountered during a (quick) build. Depending on when the error occurred, this temporary ROM may be usefully reusable, as it could be the case that some tools were inserted successfully while others weren't. It would then be a waste to discard these successful insertions just because some other tool encountered an error.

The speedup from this would only apply when fixing errors and even then it would only occasionally be noticeable, thus this is probably not worth implementing at the moment.

I'm not sure what exactly caused this, but I've got a user report of their shared palettes being corrupted while using Lunar Helper's Quick Build function to repeatedly insert sprites, custom blocks, etc. but without ever having knowingly altered their shared palettes.

The corrupted file is attached, it differs from the non-corrupted version only in the first 0x300 bytes, which is interesting and may be useful in figuring out what exactly went wrong. Don't yet know if this is some kind of Lunar Magic bug or if it could be Lunar Helper/Monitor related, but the former seems more likely to me given that Lunar Helper and Monitor only read/write the shared palette file through Lunar Magic (except for dependency analysis, I suppose).

CorruptedPalette.zip

I'm very curious about what an optimal order for insertion might be or whether there is actually a unique one. It might be the case that different hacks would benefit from different insertion order, which would be relatively annoying.

Apparently an argument can be made for inserting GPS after map16 because GPS can specify acts like settings for blocks in its list and apparently those are currently potentially overwritten by the map16 data afterwards. Not sure if this is a question of preference, I've personally never used this GPS feature so I wouldn't really know.

Personally, I think it might actually be good to insert patches at the very end of the build process since they're the most general and can potentially operate on data anywhere in the ROM, which to me implies that anything else should be inserted first so that patches can have the maximum amount of utility.

Might be a bit of a stretch, but could dramatically speed up the build process. I assume most (all?) tools probably write to different areas of the ROM, so perhaps this isn't completely impossible, though given most (all?) of them probably use freespace assigned by asar that's probably gonna be a problem.

PIXI and LM 3.31 have an incompatibility that somehow seems to prevent (some?) sprites from spawning under some circumstances. Unfortunately it seems that the way Lunar Helper works falls under those circumstances.

Not sure how to work around this, applying PIXI after everything else (including levels, overworld, etc.) did not seem to help, sadly.

Sadly, Lunar Magic is not really designed to be used on the command line and thus will sometimes throw up message boxes when something goes wrong during its parts of the build, which is not great, because:

• it's annoying
• it can actually probably make invocations like LunarHelper.exe --build halt forever

Options, sorted either from most likely to work to least likely to work or the other way around (open to interpretation):

• ask FuSoYa to do something about it
• use a timeout when invoking Lunar Magic, kill it if it takes too long, little iffy cause it does take a while to import levels if there are a lot and we don't really wanna wait that long
• change Lunar Monitor so it launches Lunar Magic instead of the other way around, passing along command-line arguments and hooking message box functions so they just use stderr or whatever instead if it's a command-line invocation (this would have additional benefits down the line probably)
• try to use Lunar Helper to hook Lunar Magic's message box functions directly, apparently this actually might work purely in C# (ok, maybe some dll imports, see https://stackoverflow.com/a/35861395/6875882)

So you can include all patches that match a pattern.

• Asar DLL
• PIXI DLL
• Human Readable Map16 DLL

Apparently Lunar Magic does not consider it a failure when you try to import multiple levels and none/some of them fail to insert.

Probably can get away with just a string check for "failed to insert into the ROM." or something like that and treating it as the failure that it should be in Lunar Helper.

Could be a very useful structure to have access to in order to determine which resources actually need to be re-inserted. Would also mean that an explicit shared folder would no longer be necessary, since Lunar Helper could infer which tools use a shared resource on its own.

Ideally, we'd determine which files the tool is going to insert and then scan those files for incsrc and incbin and build a dependency graph from that. This would require actually parsing each tool's list files and also determining which other files get inserted, i.e. everything(?) in PIXI's asm folder.

Creating a initial_patch.bps is relatively cumbersome, maybe it'd be nice to provide two already usable bps patches:

• 4MB FastROM
• 4MB SA-1

both with compression optimized for speed.

I'm aware of at least one situation where a person was using LM 3.31 with this patch. The patch requires you to reapply it after you upgrade to LM 3.31. The problem was that the current initial patches were made using LM 3.30, so basically the patch was applied on LM 3.30 and then later the levels and all other resources were inserted with LM 3.31, which essentially meant that the patch would have had to be reapplied.

This could probably also be avoided by just inserting all the LM stuff first and then running all the tools instead of doing it the other way around (see #17).

Experience seems to indicate that there is no universally ideal insertion order, which means it is not ideal for Lunar Helper to use a static one.

Open questions regarding this topic:

• Specify build order in config files or in source files?
• Specify build order explicitly, i.e. PIXI -> AMK -> UberASM, or implicitly, i.e. file A depends on file B, file A is inserted by PIXI, file B by UberASM Tool, thus UberASM Tool -> PIXI is the required order
• Logistics of specifying order in config files? Config files are parsed in arbitrary order and there's also profiles to think about.
• Form of specifying order in config files? Post-fix optional order specification via a number maybe? pixi_path = "pixi.exe" (1) to make PIXI insert first, for example
• What to do in Quick Build case? Order alone doesn't show dependencies and re-inserting everything would just take us back to square one...

All in all I think it's a complex but important topic, still not sure how to best do this while making it easy to understand and preserving Quick Build's speed.

Very open to thoughts and suggestions.