Comments (8)
If you can provide a reproducable example I can investigate it further.
from hiredis-py.
Not able to reproduce in smaller code. Can you let me know any method to check exact cause of memory leak. My problem is that when I use hiredis, the same process uses 4 times as much memory. It does not 'leak' as such but memory usage is wayyy higher.
Let me know methods to debug and I will get back with some results.
from hiredis-py.
You could try to run it through valgrind.
Without a test case it will be hard for me to track this down. Maybe you can construct a dataset of the same size? If not, and if possible, I could privately deal with your dataset in order to reproduce it?
from hiredis-py.
I found issue which is very specific to the way I implemented. I use 'signal' module to monitor if the redis fetch is completed within 10 secs. If the redis does not give response within 10 sec (which sometimes it does not because i fetch GBs of data in one query), I just raise exception and then catch it to bypass that code execution and move on to next query.
Now, when the execution is normal and function is not taking 10 sec, everything works well. If any query takes more than 10 sec and exception is raised, the hiredis implementation leaks that memory. It does not reclaim the memory while normal redis comes back to normal usage.
Does this explanation ring any bell? My dataset is very simple. I store around 15kb of binary data with 'SET' method against a key. I have around 0.3 million such key-value pairs.
I think it is not dataset but my implementation specific. I should also mention I run everything inside docker containers.
from hiredis-py.
How are you cancelling requests? It sounds like, once your timeout is reached, you just ignore the result? In which case it might very well happen that hiredis still processes the response, but as it is never returned it just leaks (whereas the pure-Python implementation will eventually garbage-collect it)
from hiredis-py.
I don't cancel, I just ignore the result. Please check sample code
def signal_handler(signum, frame):
raise Exception("Timed out!")
signal.signal(signal.SIGALRM, signal_handler)
while True:
try:
# get 'ids' from message queue
signal.alarm(10)
data = redis.mget(ids)
signal.alarm(0)
# do some stuff with 'data'
except Exception as e:
# this exception is raised when the mget does not give response in 10s
print("Timeout of 10s occured")
from hiredis-py.
That indeed looks like a bad way of using alarms. Nothing aborts the in-process requests, meaning something is likely stuck in some inconsistent state, holding on to already read memory.
You should better look for read timeouts implemented on the library than to use signal handlers here.
That is not a bug in hiredis(-py)
(sorry, mentioned Ruby in the comment above before, I of course meant Python)
from hiredis-py.
from hiredis-py.
Related Issues (20)
- Release for supporting Python 3.10 HOT 9
- Support newest hiredis release 1.1.0
- Source tarballs unavailable on pypi? HOT 5
- Changelog entry missing for 2.1.0
- Support dispatch/manual and tagged releases
- Windows wheels fail to build due to ssize_t redefinition HOT 2
- CI needs to properly account for all buildwheels attempts - at least in some labelled cases HOT 1
- sdsalloc.h doesn't seem to be installed as part of regular hiredis HOT 9
- Support garbage collection in Reader HOT 1
- Tag mismatch for 2.2.3/2.3.0 release HOT 2
- tests/test_asyncio/test_connect.py hangs on Python 3.12
- Release 2.3.0 not available on PyPI HOT 4
- Python 3.12 wheels HOT 2
- Memory leak on hgetall with RESP3 and hiredis HOT 2
- hiredis 2.3.2 probably causes fastapi app crashes HOT 1
- Hiredis cannot be installed on Python 3.13 HOT 1
- CPU spike in python:3.12.2-slim-bookworm, almost Debian image
- Segmentation fault with RESP3 map inside set
- Hiredis Is Failing To Handle `notEnoughData` Keyword Argument
- Return type differs between RESP2/3 for set operations, with hiredis HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hiredis-py.