fragglet / wadptr Goto Github PK

View Code? Open in Web Editor NEW

13.0 5.0 0.0 16.21 MB

WAD compressor

Home Page: https://soulsphere.org/projects/wadptr

License: GNU General Public License v2.0

Makefile 1.60% C 88.32% Shell 2.15% Roff 7.32% HTML 0.60%

compression doom utility wad

wadptr's Introduction

"You should have used WADPTR" - unhappily dshrunk fraggle

wadptr's People

Contributors

Stargazers

Watchers

wadptr's Issues

Upper and lower textures don't matter for one-sided linedefs

This could be used to make sidedef packing more effective

Packed sidedef scrolling walls bug

Discussed here on Doomworld.

If multiple scrolling linedefs share a sidedef, the normal scrolling speed gets multiplied by the number of linedefs because of Doom's internal data structures. It should be possible to avoid this by flagging sidedefs used by a scrolling linedef and excluding them from merge.

This should cover the original 48 scroll left type, plus also the Boom 85 scroll right type and 255 scroll-by-offsets type:

https://github.com/fragglet/mbf/blob/48b629dda9e1fc07c035ce5558fc4cd67b86f2ff/p_spec.c#L2672

Failure on EGYPT.WAD

I'm running wadptr through doomkid's "Big Vanilla Wad Pack"; 2.2 GB of vanilla and limit-removing WADs. You'll probably want to download this yourself and run through it -- just search inside the folder for EGYPT.WAD
https://www.doomworld.com/forum/topic/114671

Adding: PLATFORM       Squashing Column 0 overruns the end of the lump with no 0xff terminating byte

Even though I've seen this error message before, this is using the latest binary so perhaps is a different root cause?

WAD increases in size - Planisf2.wad

https://www.doomworld.com/forum/topic/61701

Adding: D_RUNNIN       Storing (0%), done.
Adding: MAP01          Empty (0%).
Adding: THINGS         Storing (0%), done.
Adding: LINEDEFS       Deferred... (0%)
Adding: SIDEDEFS       Packing (-1%), done.
Adding: VERTEXES       Storing (0%), done.
Adding: SECTORS        Storing (0%), done.

Merging identical lumps... done.
*** __planisf2 - Copy.wad is -8910 bytes smaller (0%) ***

Notice how SIDEDEFS packing goes to -1%!

File sizes:

original: 1,564,389 bytes
modified: 1,573,299 bytes

Failure on MULTIVERSE.WAD

https://www.doomworld.com/forum/topic/115093-e1m1-multiverse-release-24/
https://www.doomworld.com/idgames/levels/doom2/Ports/m-o/multiverse

Adding: LINEDEFS       Deferred... (0%)
Adding: SIDEDEFS       PackingRebuildSidedefs: LINEDEFS lump (#1244) is 4634 bytes, not a multiple of 16

Short blockmap lumps cause error exit

https://www.doomworld.com/idgames/levels/doom2/Ports/d-f/entropy

Adding: BLOCKMAP       Stacking BLOCKMAP lump too short: 0 < 4 header size

Sidedef packing can produce a negative compression size (???)

Also discovered when investigating #5. If sidedef packing doesn't produce a smaller result than the original, we should just use the original.

$ ./wadptr -c evimap32.wad
Adding: MAP32       	Storing (0%), done.
Adding: THINGS       	Storing (0%), done.
Adding: LINEDEFS       	Deferred... (0%)
Adding: SIDEDEFS       	Packing (-2%), done.
Adding: VERTEXES       	Storing (0%), done.
Adding: SEGS       	Storing (0%), done.
Adding: SSECTORS       	Storing (0%), done.
Adding: NODES       	Storing (0%), done.
Adding: SECTORS       	Storing (0%), done.
Adding: REJECT       	Storing (0%), done.
Adding: BLOCKMAP       	Stacking (0%), failed.

Merging identical lumps... done.
*** evimap32.wad is 0% smaller ***

Failure on Beta64.wad

This is a WAD for DOOM 64 EX:
https://www.doomworld.com/forum/topic/102849-beta-64-a-31-level-megawad-for-doom64-ex/

Adding: MAP32          Empty (0%).
Adding: THINGS         Storing (0%), done.
Adding: LINEDEFS       Deferred... (0%)
Adding: SIDEDEFS       PackingRebuildSidedefs: SIDEDEFS lump (#3) is 12768 bytes, not a multiple of 30

Option for default handling of IWADs

For automation purposes, an unexpected input prompt is not good. WadPtr will ask confirmation for IWADs. Does quiet mode (-q) remove this and what default does it take? (presumably, "no") I'd like an option to force this to "yes", e.g. -y so that all IWADs are processed without confirmation.

Corruption when uncompressing multiple files

wadptr -u *.wad and for d in *.wad; wadptr -u $d; done don't produce the same result, and the resulting WADs seem to get corrupted BLOCKMAP lumps.

Verbosity levels

Some tools allow controlling the verbosity level by a number following the -v switch. I'd like such a feature because when logging to a text file, you don't want the percentage progress, just the lump names. Secondly, some users may want an option where only the final result is shown and not the list of lumps.

Support for extended limits

Boom and other source ports support extensions to the SIDEDEFS and BLOCKMAP limits to make use of the full 16-bit range (vanilla uses signed integers, only giving 15-bits). So we should add options to support these.

Can overflow SIDEDEFS limit when decompressing huge levels

An example level is Eviternity MAP32, which is close to the SIDEDEFS limit even when they are compressed (59483 sidedefs). wadptr will happily decompress the level and generate a corrupted map. We should add the appropriate checks to check we never overflow, like we do when decompressing BLOCKMAPs.

$ ./wadptr -u evimap32.wad
Adding: MAP32       	Storing, done.
Adding: THINGS       	Storing, done.
Adding: LINEDEFS       	Deferred...
Adding: SIDEDEFS       	Unpacking, done.
Adding: VERTEXES       	Storing, done.
Adding: SEGS       	Storing, done.
Adding: SSECTORS       	Storing, done.
Adding: NODES       	Storing, done.
Adding: SECTORS       	Storing, done.
Adding: REJECT       	Storing, done.
Adding: BLOCKMAP       	Unstacking, failed.
Some BLOCKMAP lumps could not be unstacked because the decompressed
version would exceed the vanilla BLOCKMAP size limit.

$ deutex -wadir evimap32.wad
i AA17 Doom directory is /home/fraggle/doom/doom2-1.9
i WR19 Reading WAD ./doom.wad:	(2306 entries)
i WR19 Reading WAD evimap32.wad:	(11 entries)
i ID10 PWAD entry identification...

Listing of WAD directory for evimap32.wad

Entry	      Size	Type

MAP32            0	Level Map 32
THINGS       24200	Level Map 32
LINEDEFS    845250	Level Map 32
SIDEDEFS   3340890	Level Map 32
VERTEXES    203224	Level Map 32
SEGS             0	Level Map 32
SSECTORS         0	Level Map 32
NODES      3733415	Level Map 32
SECTORS     187382	Level Map 32
REJECT           0	Level Map 32
BLOCKMAP    205490	Level Map 32
i AA99 Normal exit

$ python3
Python 3.11.5 (main, Aug 29 2023, 15:31:31) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 3340890 // 30
111363
>>>

Failure on btsx_e1.wad

Note that this is specifically v0.99i9, no longer available(?), error does not occur on latest v1.6. I have attached the WAD.

Adding: WIP2           Squashing Column 7 overruns the end of the lump with no 0xff terminating byte

btsx_e1.zip

Block lists should be written in order of decreasing size

commit 3ddf4b7 added this optimization to graphic squashing and we can do the same thing for BLOCKMAPs to make prefix matching work better

BLOCKMAP compression

I wasn't aware of this when I wrote wadptr, and it can save quite a lot of space in large levels. Some nodebuilders already do it automatically.

Failure on DAEDALUS.WAD

https://www.doomworld.com/idgames/themes/TeamTNT/daedalus/daedalus

Adding: BLOCKMAP       Stacking BLOCKMAP lump too short: 0 < 4 header size

Performance could be improved for very large maps

I assumed CPUs were fast enough that this wasn't a huge issue nowadays, but when investigating #5, it really took a looooong time trying to compress Evi MAP32. Ultimately we are using an O(n²) algorithm and this can be improved.

Failure on 32in24-16_v1.wad

https://www.doomworld.com/forum/topic/91306
https://www.doomworld.com/idgames/levels/doom2/deathmatch/Ports/megawads/32in2416

Adding: SIDEDEFS       PackingRebuildSidedefs: LINEDEFS lump (#99) is 6594 bytes, not a multiple of 16

(This is with updated binary from the latest commit)

"Failed to open .wadptr-temp-000 for writing."

I get this error every time I try to use wadptr; Windows 10 x64. Doesn't matter which wad I'm trying to compress, it always fails this way.

The -o (output file) option seems to be broken

Hexen map format support

Possible to re-enable packing for special lines?

While I think that not packing special lines is good default behaviour, this can also be used to get some neat effects, so it would be nice if there were a switch to allow them to be packed, and the level designer can adjust their map to accommodate this. I'm not aware of any other currently-maintained sidedef compression tools for Doom or other convenient/automated ways to do this, and it would be a bit silly to have to keep two versions of wadptr around.

Failure on Cryogenics.wad

https://www.doomworld.com/forum/topic/98149-cryogenics-a-small-collection-of-random-cl9-slaughterish-maps-now-on-idgames/
https://www.doomworld.com/idgames/levels/doom2/Ports/a-c/cryogenics

Adding: MAP01          Empty (0%).
Adding: THINGS         Storing (0%), done.
Adding: LINEDEFS       Deferred... (0%)
Adding: SIDEDEFS       Packing (62%), done.
Adding: VERTEXES       Storing (0%), done.
Adding: SEGS           Empty (0%).
Adding: SSECTORS       Empty (0%).
Adding: NODES          Storing (0%), done.
Adding: SECTORS        Storing (0%), done.
Adding: REJECT         Empty (0%).
Adding: BLOCKMAP       Stacking BLOCKMAP lump too short: 0 < 4 header size