Code Monkey home page Code Monkey logo

Comments (6)

freeekanayaka avatar freeekanayaka commented on August 23, 2024

Is it possible to get some insight about what data those users store into their DBs to make them grow as large as a GB? Gut feeling is that there might room for some mix approach where part of the data gets stored outside dqlite.

Regarding the compression approach, do you have access to sample snapshots and have an idea of how much they would compress with (say) lz4?

from raft.

MathieuBordere avatar MathieuBordere commented on August 23, 2024

Depending on the speed of the compression chosen, they compress between 8/1 and 12/1 ratio which is pretty good. It's just 1 particular microk8s user that I know of that has this issue, but there are/will be probably more, not sure if they can store data elsewhere.

from raft.

freeekanayaka avatar freeekanayaka commented on August 23, 2024

Depending on the speed of the compression chosen, they compress between 8/1 and 12/1 ratio which is pretty good. It's just 1 particular microk8s user that I know of that has this issue, but there are/will be probably more, not sure if they can store data elsewhere.

I see. That's a good compression ratio, if that would be enough to solve this particular issue, it's probably ok. However I guess it will only become clear once implemented. Maybe a more generic solution would be to introduce some way to "throttle" the speed at which snapshot are being written, so they don't disrupt too much the normal flow of raft entries.

from raft.

MathieuBordere avatar MathieuBordere commented on August 23, 2024

Throttling the speed could also work. Luckily for me, I have a crappy ssd drive that starts to show that behavior after a while, I'll try both approaches and see which one appears to work best.

from raft.

freeekanayaka avatar freeekanayaka commented on August 23, 2024

Throttling the speed could also work. Luckily for me, I have a crappy ssd drive that starts to show that behavior after a while, I'll try both approaches and see which one appears to work best.

FWIW throttling could be done by writing the snapshot in chunks (as opposed to a single pass) and waiting a bit between each chunk write. Ideally how much to wait would be decided by taking the current "load" of the disk into account, but probably some hard-coded value like a few hundred milliseconds might work ok.

The advantage of throttling is that you don't have to change the on-disk format and also you should be able to scale better (say you have a 10GB disk, then compression might stop helping). Of course the downside is that it might be a bit more complex to implement.

from raft.

MathieuBordere avatar MathieuBordere commented on August 23, 2024

While introducing compression I will also tackle #201, lz4 has an option to perform a crc check.

from raft.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.