Memory leak when querying large dataset about hiredis-py HOT 8 CLOSED

If you can provide a reproducable example I can investigate it further.

from hiredis-py.

Not able to reproduce in smaller code. Can you let me know any method to check exact cause of memory leak. My problem is that when I use hiredis, the same process uses 4 times as much memory. It does not 'leak' as such but memory usage is wayyy higher.

Let me know methods to debug and I will get back with some results.

from hiredis-py.

You could try to run it through valgrind.
Without a test case it will be hard for me to track this down. Maybe you can construct a dataset of the same size? If not, and if possible, I could privately deal with your dataset in order to reproduce it?

from hiredis-py.

I found issue which is very specific to the way I implemented. I use 'signal' module to monitor if the redis fetch is completed within 10 secs. If the redis does not give response within 10 sec (which sometimes it does not because i fetch GBs of data in one query), I just raise exception and then catch it to bypass that code execution and move on to next query.

Now, when the execution is normal and function is not taking 10 sec, everything works well. If any query takes more than 10 sec and exception is raised, the hiredis implementation leaks that memory. It does not reclaim the memory while normal redis comes back to normal usage.

Does this explanation ring any bell? My dataset is very simple. I store around 15kb of binary data with 'SET' method against a key. I have around 0.3 million such key-value pairs.

I think it is not dataset but my implementation specific. I should also mention I run everything inside docker containers.

from hiredis-py.

How are you cancelling requests? It sounds like, once your timeout is reached, you just ignore the result? In which case it might very well happen that hiredis still processes the response, but as it is never returned it just leaks (whereas the pure-Python implementation will eventually garbage-collect it)

from hiredis-py.

I don't cancel, I just ignore the result. Please check sample code

def signal_handler(signum, frame):
    raise Exception("Timed out!")

signal.signal(signal.SIGALRM, signal_handler)

while True:
    try:
        # get 'ids' from message queue
        signal.alarm(10)
        data = redis.mget(ids)
        signal.alarm(0)
        # do some stuff with 'data'
    except Exception as e:
        # this exception is raised when the mget does not give response in 10s
        print("Timeout of 10s occured")

from hiredis-py.

That indeed looks like a bad way of using alarms. Nothing aborts the in-process requests, meaning something is likely stuck in some inconsistent state, holding on to already read memory.
You should better look for read timeouts implemented on the library than to use signal handlers here.

That is not a bug in hiredis(-py)

(sorry, mentioned Ruby in the comment above before, I of course meant Python)

from hiredis-py.

from hiredis-py.