Add support for etcd about limits HOT 5 CLOSED

From a very basic evaluation I think only fixed window rate limiting might be achievable (which is fine) though it's still not clear to me how one could atomically increment a value.

I've personally only used etcd for simple distributed config and never for anything involving counters or collections (needed for the moving window strategy), so I'm probably not the best person to try and implement this feature.

Having said that, if you have some pseudo code examples how one might implement the strategies that limits supports, I'm happy to give it a crack myself.

from limits.

Looking at some stuff, I think you could do something like:

lock = client.lock(b'rate-limit-key')

await lock.acquire()

key = await client.get(b'rate-limit-key')

if key is None:
   await client.put(b'rate-limit-key', value)
else:
   await client.delete(b'rate-limit-key')
   await client.put(b'rate-limit-key', value)

await lock.release()

The only problem I could see with this is that it uses locks, because etcd doesn't support atomic increments itself and any alternatives/fixes seem too complicated, especially in async or distributed environments.

from limits.

I've pushed a very basic implementation of fixed window / fixed window + elastic expiry with etcd for both sync/async (using python-etcd3 and aetcd).

I ended up using transactions instead of locks since that made more sense to me. Leases don't fully satisfy the requirements of implementing rate limits since a lease doesn't guarantee expiry at an exact time so some weird hacks had to be done to encode the expiry into the value stored in etcd instead of purely relying on the ttl returned by the lease (one non trivial example is trying to set a rate limit of n/1 second which won't work by depending only on a lease to expire the counter in a default etcd deployment since the minimum lease duration is 2 seconds).

Official support for this might not happen immediately since I noticed both python-etcd3 and aetcd are not very well maintained and require some workarounds to consistently install across python 3.8-3.11.

Regardless, this is available in master now and I'll do a release soon that includes this.

from limits.

Available now in 3.0.0

from limits.

@VincentRPS upon running some benchmarks I realized the etcd implementation is much slower than the redis/memcached/mongodb counterparts - approximately 10x in some cases. These benchmarks are run on an untuned single node docker container (and I understand etcd works best when backed by high performance disk), but I'm wondering if there is any way to configure etcd in a development environment for testing to emulate optimal performance (maybe a ram disk or something?) or if there are some obvious tweaks to the code that can help (though even a simple get operation in my benchmarks is significantly slower).

from limits.