Comments (6)
Is it possible to get some insight about what data those users store into their DBs to make them grow as large as a GB? Gut feeling is that there might room for some mix approach where part of the data gets stored outside dqlite.
Regarding the compression approach, do you have access to sample snapshots and have an idea of how much they would compress with (say) lz4?
from raft.
Depending on the speed of the compression chosen, they compress between 8/1 and 12/1 ratio which is pretty good. It's just 1 particular microk8s user that I know of that has this issue, but there are/will be probably more, not sure if they can store data elsewhere.
from raft.
Depending on the speed of the compression chosen, they compress between 8/1 and 12/1 ratio which is pretty good. It's just 1 particular microk8s user that I know of that has this issue, but there are/will be probably more, not sure if they can store data elsewhere.
I see. That's a good compression ratio, if that would be enough to solve this particular issue, it's probably ok. However I guess it will only become clear once implemented. Maybe a more generic solution would be to introduce some way to "throttle" the speed at which snapshot are being written, so they don't disrupt too much the normal flow of raft entries.
from raft.
Throttling the speed could also work. Luckily for me, I have a crappy ssd drive that starts to show that behavior after a while, I'll try both approaches and see which one appears to work best.
from raft.
Throttling the speed could also work. Luckily for me, I have a crappy ssd drive that starts to show that behavior after a while, I'll try both approaches and see which one appears to work best.
FWIW throttling could be done by writing the snapshot in chunks (as opposed to a single pass) and waiting a bit between each chunk write. Ideally how much to wait would be decided by taking the current "load" of the disk into account, but probably some hard-coded value like a few hundred milliseconds might work ok.
The advantage of throttling is that you don't have to change the on-disk format and also you should be able to scale better (say you have a 10GB disk, then compression might stop helping). Of course the downside is that it might be a bit more complex to implement.
from raft.
While introducing compression I will also tackle #201, lz4 has an option to perform a crc check.
from raft.
Related Issues (20)
- missing getrandom on centos 7 HOT 1
- [question] Usage of raft_add, raft_remove, and raft_assign HOT 2
- [question] When is the first leader elected? HOT 2
- recvAppendEntries: Assertion `r->state == RAFT_FOLLOWER || r->state == RAFT_CANDIDATE` failed HOT 9
- [question] Forwarding request to leader HOT 8
- Potential use-after-free in handling of raft_transfer HOT 5
- Fix the 32-bit CI HOT 1
- can't build on m1 mac (`<linux/xxxxx.h>` is missing) HOT 3
- v1.x RFC: pull based approach HOT 5
- raft_start(): io: closed segment xxx is past last snapshot xxx HOT 4
- [question] Understanding how to add new servers HOT 15
- ./configure fails when no external dependency is found HOT 4
- Assertion: src/uv_truncate.c:168: UvTruncate: Assertion `index < uv->append_next_index' failed. HOT 3
- src/replication.c:457: getRequest: Assertion `req->type == type' failed. HOT 3
- Segment writes blocked when taking a snapshot. HOT 3
- src/log.c:87: refsTryInsert: Assertion `next_slot->term != term' failed. HOT 9
- Jepsen: Another truncate-related assertion failure
- [suggestion] add lock/unlock function pointers to raft_io (and use them) HOT 1
- `uvOsFallocateEmulation` taking too long time HOT 1
- CI: `xfs` test failures HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from raft.