Increase A-Life ID limit about xray-16 HOT 5 OPEN

u32 max value is 4 294 967 295. The A-Life ID limit is not even u32, it's u16, hence the limit of 65535 (max value of u16) entities.
So, increasing it to u32 will practically fill all the needs of modmakers.

That will require some work, though.
We will need to find all usages of u16 related to the ID system and replace that with u32.

from xray-16.

from xray-16.

Note: This is a known chronic problem for mods with larger game worlds, especially Anomaly modded with very high "war-zone-like" or "zombie-apocalypse-like" spawn rates, such that some hours into a game it becomes unplayable due to frequent crashes. Which suggests three things, (1) if it were "low hanging fruit" of an improvement it would have been done already so it maybe more complicated than expected, (2) garbage collection/clearance is also a problem such that once the cap is raised then runaway memory usage may follow, and (3) once done this has significant value as a promotional item able to attract more community attention via demoing truly ridiculous game world populations (e.g. may encourage more help in completing Anomaly support). There maybe useful discussions on this in Anomaly and Anomaly mod forums and issue trackers. Hope that helps, sorry I can't be of more help directly.

from xray-16.

(1) if it were "low hanging fruit" of an improvement it would have been done already so it maybe more complicated than expected

Yes, there are many places where u16 is used, so we need to thoroughly analyze the engine code.
One person told me that game scripts also written with u16 in mind, so we need rewrite not only the engine parts, but also the Lua part.

(2) garbage collection/clearance is also a problem such that once the cap is raised then runaway memory usage may follow

The engine architectural solution for releasing objects references in Lua is very suboptimal. It triggers FPS drops -> lag spikes.
So, introducing bigger ID limit is only the first step. Then, we would need to even more rewrite the engine code to make it optimized, i.e. make the game playable with the bigger number of entities.

There maybe useful discussions on this in Anomaly and Anomaly mod forums and issue trackers.

It would be great, if you or someone could hint us with the links the these discussions.

from xray-16.

Last time I read into this was early 2022 and I'm having trouble finding some stuff I remember seeing. There seems to be very little practically useful technical discussion, but bug reports thoroughly establish there is a problem and skimming over the player oriented threads helps to get a feel for the scope of the problem.

Here are two decent Reddit threads:

• A "short" guide to handle A-Life issues and crashes related to total item count
• We need group ideas to resolve the object number limit problem, don't miss the hording meme worthy screenshot of ~20k rounds of ammo dropped.

Here are two older Anomaly 1.5.0 threads:

• 65535 objects and many related crashes, with an overview and references older crash reports.
• 65535 objects part 2, with a graph of object ID growth over time.

I did a few searches on the Anomaly bug tracker, haven't found anything useful yet, seem to remember something useful being here.

Need to dig through other mods forums and any trackers.

Some notes on those threads and understanding the problem:

• Object ID's are issued to anything active in the game world, like Stalkers, mutants, circling crows, movable & destroyable props & vehicles, and inventory items carried by those characters, dropped on the ground, held by merchants, or kept in storage containers.
• Comments claiming ID's are not recycled are inaccurate but it's complicated. If an object is properly despawned the engine will reuse the ID (e.g. dropped items cleaned off the map), it's unclear if this has much leakage. For some reason the ID's of items removed by scripts are more often not released for reuse, and that "leakage" builds up overtime to deplete the 65k ID's available.
• Player item hording is a major cause of object ID inflation, they can amass tens of thousands of items scattered across numerous stashes, and this gets significantly worse as more types of items are added, especially entire categories of new items.
• Among modders there is a general creative push toward fully realizing everything with dynamic objects, leading to mods of mods which add many more types of inventory items, like large weapon & armor packs with weapon & armor stripping to parts, weapon & armor upgrades as kit items, numerous types of repair kits, crafting of kit items, new ammo calibers, specialty ammo types (multiplier to calibers), ammo magazines, other crafting, more realistic sets of mutant parts, cooking, making damn near everything destroyable, etc, all of which causes significant object ID inflation.
• Among players there is a general push toward more "realistic" gameplay and larger game world populations, it's a bad combination with this ID limit.
• Developers of large mods have mostly worked within the 65k limit by limiting the amount of "stuff" spawned into the game world and cleanly removing it, but mods of those mods tend to be sloppier with managing items so they are not always removed in a way the ID can be released for reuse, causing significant ID inflation over a long game. It may also be lua functions do not work properly in certain situations, so ID's are not released when they should be.
• There are theories about creating a utility to "clean" save game files of unused ID's, I'm not aware of any existing tools, but this could be built into the engine to run during save game loading when performance issues due to high CPU load are not disruptive.
• High spawn rates of Stalkers is a problem because of their inventories but the same is not true of mutants. For example, if each Stalker carries two dozen items then a population increase from 1000 to 3000 would inflate object ID's from 25k to 75k (e.g. war zones & zombie apocalypse), but mutant inventories are usually empty with loot spawned upon "harvesting" (this differs from mod to mod).
• Mod scripts can readily track ID's of objects they spawn and clean up after themselves when those objects are no longer needed, but it's clunky, kind of redundant, not Alife managed, and leaks.
• Scripts may intentionally remove items while intending later use, especially quest scripts, so aggressively releasing apparently unused ID's for reuse may cause problems. A means to give ID's protected status from purging is probably necessary.

Suggestions:

• Change object ID's from 16bit to 32bit, could analyze this in depth first, or just create a new branch to see what breaks & how then iterate fixes from there.
• Add soft ID cap as a game setting to throttle ID inflation, adjust aggressiveness of ID culling when over this limit.
• Change the lua function behavior to add the ID's of removed items to a purge queue, with a game time stamp, this group can be more aggressively purged.
• Add a more CPU intensive ID usage check routine during save game loading to build a second purge queue of apparently unused ID's, and purge items which have been in a queue for more than a certain amount of game time.
• Add a protected ID list and simple algorithm to avoid purging things which are used intermittently, probably based on game time duration of protection, modders could use a function to permanently protect certain things used intermittently or for later use in quests, and always protect things with certain properties (e.g. quest items).
• Improve item stack handling, so identical items inside of the same container always have a single ID with a quantity value. This should have a small improvement to large player stashes and merchants with large inventories, and a tiny improvement to ID inflation from all NPC inventories.
• Add stash inventory abstraction, so when an offline map is cleaned up the standard item contents of any static containers are replaced with an inventory list attached to the container object properties, so only items with special properties (e.g. modded guns & armor) or special (e.g. quest) items remain as normal, then restore the inventory when the container is opened. NPC item seeking and usage behavior may need adjusted to use that abstracted inventory list too. This should decrease the rate of ID inflation from player hording across numerous stashes, but probably won't help much for frequently accessed player "home" stashes. Something similar could be done for infrequently visited merchants, but this may break things like larger economic simulation and NPC's really trading with merchants (I don't know if any mods do this but it would be cool).
• Add NPC behavior to sell unneeded loot, simply dumping items on the closest merchant creates a sort of gameplay accessible purge queue and more dynamic feel to trading.
• Consider abstracting ammunition items, for examples:
  • Replace small "boxes" with ammo stacks of any arbitrary size, each has one ID, the box size becomes a default stack size only used for item spawning, loot, purchases. I believe this could be done in a mod agnostic fashion.
  • Add a large ammo storage container feature, to fully abstract out ammo from regular inventory, with a single object ID. To go with it, add a few related features:
    • Types of ammo "cans", NATO style bulky metal boxes with good protection, light plastic boxes with minimal protection, wood boxes with a balance, waterproof cloth bags.
    • Hardcore environmental hazards for carried ammo, including water damage, anomaly damage, fire & explosive cook off. Warsaw pact style "span cans" could be the only small box with good environmental protection.
    • For trade, designated player stashes could come with a free ammo can, and merchants can stock bulk military issue ammo in cans and sell various empty cans.
    • Add an ammo inventory management menu with various useful features, and a hook system for mods to easily add more. For examples, toggle auto adding picked up ammo to the container, set desired inventory stock level for each ammo type, designate a second container for surplus ammo (loot dump bag), speed up loading magazines (and draw from loot container first), easier ammo disassembly, easier ammo hand loading, etc.
    • It would be up to mod developers to fully realize this in-game.
  • Abstract NPC ammo so it's a property of the NPC object, then convert to inventory items only for trade and loot.
• Note, ID inflation is still a significant performance issue, but the above improvements may not be worth the effort if the underlying performance issues can be resolved.

Edit, added some more notes...

Related Issues:

• Object ID's are held in various lists in the Alife system and held as values in various scripts. A used ID needs a little time to phase out of use. If an ID is reused immediately it may end up as an entirely different kind of object while also being seen as still active so old references to it are not cleared. This is why aggressive purging of ID's could cause serious problems.
• Being near or opening containers with large numbers of items causes lag or momentary hang due to iterating through the list of ID's. This needs investigation for improvement.
• Parts of object ID management are clunky and cause overall performance slowdown when ID count nears the 65k limit. Forum discussion suggests this is due to the sheer number of active object ID's, based on saving & reloading not yielding significant improvement, but liquidating player stockpile then saving & reloading does yield significant performance improvement. Performance profiling is needed to figure out why.
• Various mods make claims of script improvements, with occasional reference to things related to object ID handling. It may help to do community outreach to have improvements contributed upstream.

from xray-16.