Comments (2)
Thanks for the detailed analysis and ideas.
Some comments:
aiosqlite
might only solve the problems partially. It seems to bundle open connections in a seperated thread to not open multiple connections to the database. This helps but might not be enough on high load and especially on poor IO or multiprocessing. In the end SQLite is not that ideal for high load but for most use cases of this library a good trade-off between performance, easy to use/availability and available and resilience to errors.
Is using the in-memory database an option for your use case?- using seperate threads for processing, storing and transmitting events might help but probably requires quite some efforts to implement properly and might still suffer from Python's Global Interpreter Lock (all threads running on the same CPU core)
PriorityQueue
was introduced in #73 to workaround high memory usage of CPython'sQueue
implementation. Though if there is another similar queue implementation which supports getting items in batches, that would probably help- to address the problem that the queue never raises the
Empty
exception on high load and so the events from the database won't get transmitted, we could set a maximum size on the queue, use.put(block=False)
and then handle theFull
exception to force flushing pending events.
I think I will work soon on the last item to give the queue a maxsize and handle a full queue accordingly as this might improve the overall behavior.
Though for the other options it's unlikely I will be able to spend much time on this, as it would probably also require refactoring lots of code for a maybe only special use case.
I'd suggest to think about using the in-memory database or at least test whether it will improve the overall performance. If you like, feel free to try to port the code to aiosqlite
, this sounds like relatively trivial and might already help a bit.
from python-logstash-async.
Related Issues (20)
- Django processes hang HOT 5
- Log record "extra" fields are missing when using LogstashFormatter HOT 6
- Log level of: "An error occurred while sending events: [Errno 104] Connection reset by peer" HOT 2
- Not working with Django HOT 5
- Crash when logstash server is down (gunicorn with gevent worker) HOT 1
- ImportError: cannot import name 'ParamSpec' from 'typing_extensions' on Python 3.11 HOT 1
- Build pipeline no longer works HOT 1
- Logstash TCP transport errors if not using a client certificate HOT 1
- Sqlite DB not vacuumed HOT 2
- ModuleNotFoundError: No module named 'fcntl' HOT 3
- Broken TCP connection and endless waiting for events from the dark HOT 2
- Makes get_queued_events FIFO or LIFO HOT 1
- Retrieve non flushed event count in sqlite db at start HOT 1
- 2.7.0 doesn't work under Windows HOT 1
- Can cause python code to hang in k8s deployment with python 3.11 HOT 4
- AttributeError: 'MemoryCache' object has no attribute 'get_non_flushed_event_count' HOT 3
- Transport endpoint is not connected HOT 10
- JSON Parse Failure on host Field (could not set field 'ip' on object) HOT 2
- BeatsTransport _batch_size HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from python-logstash-async.