Replicated adapter syncing during rolling deployment about nebulex HOT 5 CLOSED

Alright, that makes sense. I will check your suggestion anyway. Also, it is planned for the replicated adapter to use Mnesia in the near future, so keep you posted. Thanks!

from nebulex.

Everything seems to be lost on deployment. I tried with 3 instances to give more time potentially for syncing, but it didn't help.

from nebulex.

Hey!

Can it be expected that no data is lost during such a deployment with the replicated adapter?

Unfortunately, that cannot be guaranteed. As you know, a deployment can be done in many different ways, so the cache can't be aware of that. The replicated cache tries to keep consistency across the "Erlang/Elixir Cluster", however, it doesn't guarantee "no data loss".

3. Will instance 2 wait with the shutdown until the cache is in sync?

No, it won't. When you shut down an instance, you are killing the cache too, the cache is not aware you are updating or shutting down the instance; it does a clean-up on termination ensuring the stopped cache leaves the cluster, but that's all (and this will happen only if you stop the app – like using Application.stop(...) or calling your release stop script my_app stop, like a graceful shutdown). So, maybe what is happening when you update instance 1, the sync-up process may have started, but instance 2 is immediately updated, then instance 1 is not able to finish the sync-up process (or maybe it happens so fast that the sync-up hasn't started yet), means, there is no data at that point in time.

On the other hand, if "cache data loss" is critical for you, I'd take a different approach. Currently, the cache runs within your app (same supervision tree) in the same instance. There may be other approaches (a bit more complex but more robust):

• "Cache Servers" topology – cache servers running outside your app (like a separate cache app), on a separate cluster, so they are not affected by your app deployments.
• "Redis Cache" – you could use the NebulexRedisAdapter and use Redis as a cache. This is usually the approach I take for more robust and complex scenarios. Here, you can leverage Redis features, like, cluster mode, persistency, sharding topology using Envoy proxy, etc.

Anyway, please let me know if that helps, I stay tuned!

from nebulex.

Hey thanks for confirming!

Would it be in vain to do the following:
Add a callback option to the replicated adapter that is called when the sync is complete, and have that callback be tied to the healthcheck of the app (set a value in a local cache or keep it in a genserver).
Now until that value is set to true the healthcheck on the new instance would fail and keep the deployment from moving forward.

I am experimenting with this on this fork: benonymus@2f6036c

from nebulex.

We decided to use mnesia with https://github.com/sheharyarn/memento for the shared cache, thanks for the responses!

from nebulex.