Comments (6)
I've checked the official documentation of Badger regarding garbage collection. @woodzy , you're right.
Since neither of us has experience using BadgerDB in a production environment, if you could, it would be great if you could submit a PR. This would be very helpful for most users who use BadgerDB as a persistent database.
Garbage Collection
Badger values need to be garbage collected, because of two reasons:
Badger keeps values separately from the LSM tree. This means that the compaction operations that clean up the LSM tree do not touch the values at all. Values need to be cleaned up separately.
Concurrent read/write transactions could leave behind multiple values for a single key, because they are stored with different versions. These could accumulate, and take up unneeded space beyond the time these older versions are needed.
DB.RunValueLogGC(): This method is designed to do garbage collection while Badger is online. Along with randomly picking a file, it uses statistics generated by the LSM-tree compactions to pick files that are likely to lead to maximum space reclamation. It is recommended to be called during periods of low activity in your system, or periodically. One call would only result in removal of at max one log file. As an optimization, you could also immediately re-run it whenever it returns nil error (indicating a successful value log GC), as shown below.
ticker := time.NewTicker(5 * time.Minute) defer ticker.Stop() for range ticker.C { again: err := db.RunValueLogGC(0.7) if err == nil { goto again } }Memory usage
Badger’s memory usage can be managed by tweaking several options available in the Options struct that is passed in when opening the database using DB.Open.
- Number of memtables (Options.NumMemtables) If you modify Options.NumMemtables, also adjust Options.NumLevelZeroTables and Options.NumLevelZeroTablesStall accordingly.
- Number of concurrent compactions (Options.NumCompactors)
- Size of table (Options.BaseTableSize)
- Size of value log file (Options.ValueLogFileSize)
If you want to decrease the memory usage of Badger instance, tweak these options (ideally one at a time) until you achieve the desired memory usage.
from server.
@woodzy Can this #371 fix the issue?
from server.
Thanks! These look like better implementations of my local workarounds during experimentation.
from server.
This should be fixed in v2.6.0
- please let us know!
from server.
It not fix, still growing on mem
from server.
@woodzy @OpenJarvisAI Could this issue be closed?
from server.
Related Issues (20)
- After enabled badger, the vlog file up to 700M one day and 4GB one week HOT 9
- Race condition when running the redis example HOT 4
- 遍历Clients时如何判断当前Client是否为Disconected状态 HOT 3
- 作者您好,请帮忙关注一下这个问题 HOT 3
- Hi, what is the simplest way to make messages can be restored when server cut off? HOT 5
- The badge still getting vlog file keep growing infirnity HOT 7
- How to send topics posted by specific users only to specific subscribed users? HOT 11
- Does peddle perssistant released? HOT 5
- MQTTX cannot use Topic Alias. MQTT5.0 主题别名发送卡住,无法发布主题别名的消息 HOT 2
- How to use the new pessistent hook? HOT 1
- Reload auth fIle on the run HOT 2
- InlineClient模式下服务端订阅问题,inline subscribers do not receive messages HOT 5
- Merge 2 version of storm HOT 4
- Add Support for Disconnect With Will Message Reason Code
- Logging Level is not Configurable Via File Configuration
- Persistence storage did not work with SetCleanSession(false) HOT 3
- Don't allow inheriting session unless username matches HOT 5
- MessageExpiry Hook HOT 1
- OnConnectAuthenticate cannot specify an error code (like Client Identifier not valid)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from server.