Is it possible to use a Redis connection pool? about limits HOT 20 CLOSED

Awesome! Thanks for helping verify and also the idea!

from limits.

if you running this under gunicorn etc... the short answer is no, each thread will be launched under its own application context, so it wont share a GIL ...

What I personally ended up doing , was using nutcracker and having python connect to that, to minimize the impact to the redis server

Edit:

ahhh sorry i came back to edit this, i just remember, due to limits package running "script" cmds to redis i couldnt run limiter through nutcracker

Edit 2:

I haven't tested it yet... but you could try https://github.com/RedisLabs/redis-cluster-proxy

from limits.

When you say connection pool, do you mean being able to do something like this:

from limits.storage import RedisStorage
from limits.strategies import MovingWindowRateLimiter
import redis
pool = redis.ConnectionPool(host='localhost', port=6379, db=0)
storage = RedisStorage(connection_pool = pool)
limiter = MovingWindowRateLimiter(storage)

Unfortunately not at the moment - since the redis implementation delegates the construction of the redis client entirely to the redis.Redis.from_url function.

Having said it does seem reasonable to reuse a shared connection pool in your application and there have been other use-cases where letting the application provide a "Storage" that has been pre-constructed.

from limits.

This issue should be in the limits repository, so I'll transfer it there.

from limits.

Well the problem comes when your using flask / gunicorn for example under green threads.. gunicorn isnt going to share the connection / connection pool because of the green thread...

So each thread is going to open up its own connection pool and when you have a cluster of servers it really hits the redis... when you doing thousands of requests a second

you can put nutcracker on the local system as a "middleware" to use just one connection pool from the server itself to the redis server , but due to the exact queries your code is running, it isnt supported in nutcracker

https://github.com/twitter/twemproxy

from limits.

I think you might be able to use redis-proxy in envoy (which supports lua scripting afaik) reference

from limits.

When you say connection pool, do you mean being able to do something like this:

from limits.storage import RedisStorage
from limits.strategies import MovingWindowRateLimiter
import redis
pool = redis.ConnectionPool(host='localhost', port=6379, db=0)
storage = RedisStorage(connection_pool = pool)
limiter = MovingWindowRateLimiter(storage)

Unfortunately not at the moment - since the redis implementation delegates the construction of the redis client entirely to the redis.Redis.from_url function.

Having said it does seem reasonable to reuse a shared connection pool in your application and there have been other use-cases where letting the application provide a "Storage" that has been pre-constructed.

HI @alisaifee,

just circling back around to this, to see if this would be possible ? as if you using gunicorn etc, using a redis BlockingConnectionPool and monkey patching gevent can limit the number / rape connections being opened

would be awesome if we could just pass a redis instance to limiter rather than a URI, that way we can share an already created pool instance with the rest of the application and not create an entire new pool...

from limits.

could

be replaced with something along like the follwoing...

    redis = self.dependencies["redis"]
    if isinstance(uri, redis.client.Redis):
        self.storage = uri
    else:
        uri = uri.replace("redis+unix", "unix")
        self.storage = redis.from_url(uri, **options)

from limits.

@VeNoMouS could you take a look at #110 to see if that'll work for you?

from limits.

Will test today for you, i'll have to hack your flask limiter up first to allow the new param

mmmm i'll make some simple test code and test outside of flask first to see if it uses pool connection

from limits.

You shouldn't need to hack anything in flask-limiter, this should work:

from flask_limiter import Limiter
pool = ....
limiter = Limiter(....., storage_uri="redis://", storage_options={"connection_pool": pool})

from limits.

You shouldn't need to hack anything in flask-limiter, this should work:

from flask_limiter import Limiter
pool = ....
limiter = Limiter(....., storage_uri="redis://", storage_options={"connection_pool": pool})

ok cheers will test out in a bit and let you know

from limits.

🙇

from limits.

Since my stuff is in a docker... this is a dirty trick to pull the code base with the PR as a zip for requirements.txt incase you didnt know...

https://api.github.com/repos/alisaifee/limits/zipball/pull/110/head

from limits.

Appears to be working @alisaifee

from limits.

Testing it now on a test production server

from limits.

I've cut a 2.4.0 release if you want to give that a shot as well!

from limits.

Sure , give me ~hour and ill roll out another test

from limits.

2.4.0 appears to be working

from limits.

thanks for patching ❤️

from limits.