Ability to refer to entities by identifiers stored in an external data structure about becsy HOT 6 CLOSED

So before I dive deep into the idea of indexes and such, I have a simple question: could the tile map store entities directly, rather than just IDs? That is, you'd do something like this:

const visible = this.tileMap.addElement(
        entity.hold(), 
        entity.read(PositionComponent), 
        entity.read(SizeComponent)
      );

And then:

const visibleElements = this.tileMap.visibleElements(camera);  // returns array of entities

I think the only caveat is that you must ensure you never access deleted entities but that should be pretty easy to handle with a removed query that wipes them from the tile map. (Or perhaps these entities live forever once created in your app?)

What do you think?

from becsy.

Thanks for the quick response!

That's an interesting idea; thanks for sharing 🙏 I'd definitely like to consider .hold() as an option, I think it would help with solving the problem I have. I do think there's a few things which are constraining me in my current project, though:

• Architecturally we're trying to keep our data structures independent of our ECS. We've been applying this principle to other parts of our stack too (e.g. our rendering layer). In this specific case, we're eager to have the one virtualisation contract which can be extended and evolved over time with different strategies (e.g. the quadtree I mentioned above).
• Over time, we want to reflect the virtualisation algorithm we choose and its domain model across our stack. I.e., we hope to move some parts of our tiling implementation to the server-side parts of our app, and to share stable IDs between client and server, with the data structures speaking to one another up-and-down the stack.

More generally to the last point: Since we have server-side infrastructure and ingest events from it, we have a few other use cases where indexed access (or some form of it) would be amazing 😄 Currently, due to this event-driven infra, we need to look through all entities in the world to synchronise a local ECS with what has happened in other parts of the stack / on other machines.

E.g., if an entity is deleted by person A or via a HTTP API call, we supply an ID to person B, and then person B has to sweep through all the entities in their ECS to finally delete the correct entity (based on an IdComponent lookup).

One last question I wanted to ask you regarding .hold(): Does holding every entity have any 'bad' consequences, in your view? My first instinct was to worry a bit about performance and memory, as I know becsy does a bit of heavy lifting on the storage compaction side of things; would holding every entity have an adverse impact on these two?

Thanks again for helping me out with this - really appreciate it!

from becsy.

I think the answer to both your concerns is the same: if you want or need to design your data structures in terms of raw IDs, then you just shift the translation to a separate mapping layer where you still use .hold() to target actual entities. This could be as simple as the following, though you might need to tweak it if you require new entities to be mapped as soon as they're added rather than just once per frame.

class IdMapper extends System {
  entities = this.query(q => q.added.removed.with(IdComponent));
  map = new Map();

  execute() {
    for (const entity of this.entities.added) this.map.set(entity.read(IdComponent).id, entity.hold());
    for (const entity of this.entities.removed) this.map.delete(entity.read(IdComponent).id);
  }

  lookup(id) {
    return this.map.get(id);
  }
}

class SomeOtherSystem extends System {
  mapper = this.attach(IdMapper);
  execute() {
    // ...
    entity = this.mapper.lookup(id);
  }
}

There are no material downsides to using .hold(). As mentioned previously, you do need to ensure that you don't keep using the entity handle after the entity has been deleted, but that's fairly straightforward to deal with (especially as you always have a full frame to update your data structures). There's some overhead in dev mode for Becsy to check this invariant, but it goes away in perf mode, where .hold() just creates a new lightweight object that captures the right internal ID. I guess if you create/drop held entities often you might have a bit of GC churn to contend with at worst...

I think the only case where built-in indexing would matter is in multi-threading situations, since attaching all systems to a single, thread-specific map would tank concurrency. I haven't completely figured out how to address this yet but there are two basic approaches:

1. Replicate the map structure per-thread and use Becsy's internal "log" system to ensure that changes are processed in all threads. This is how refs fields work. It's safe, fairly easy, and has great read performance, but write performance goes down with the number of threads since each one needs to redo all update work.
2. Design dedicated shared-memory index structures that create ephemeral entity handles on access. This would be more tricky to manage and get right, and read performance would likely be a bit worse due to the entity handle allocation / borrowing overhead, but it may be worth it in some situations.

I intend to dig more into these options once I get multithreading running, since it'll be hard to experiment with stuff before then and I don't want to commit prematurely to a bad design!

from becsy.

Gotcha!

I think the approach you've described with the holder map would indeed meet the use case of our existing project 👍 Super keen to see where you land with the indexing stuff when you get to it!

As a completely random idea / out of interest: I wonder if there is some overlap with the problems you're trying to solve with multi-threaded storage and making use of a light, thread-safe DB layer such as absurd-sql or sql-lite in the browser 😄 Not sure if you've already looked into it. I'd be surprised if they give you the perf & concurrency guarantees you're after, though.

On the topic of this specific issue though - thanks, this approach should unblock us for time being!

from becsy.

Ah, I hadn't considered using IndexedDB or one of the adapters built on top of it! It does give strong concurrency guarantees, but I think you're right that performance is likely to be poor, at least in comparison to direct shared array writes. Another constraint is that Becsy needs to run both in browser and in Node, and I didn't find any good IndexedDB polyfills for the latter -- not much demand, I guess, and they especially don't address concurrency issues. Probably best to stick to shared arrays...

from becsy.

Another constraint is that Becsy needs to run both in browser and in Node

Makes sense!

If you consider looking into it, some form of sqlite wrapper may suit that need, as it can run in both a browser-based environment and Node 😄 I can't confess to having used it a huge amount, but have found it gives reasonable performance with careful schema design.

from becsy.