zjuytw / v6d Goto Github PK
View Code? Open in Web Editor NEWThis project forked from v6d-io/v6d
vineyard (v6d): an in-memory immutable data manager. (Project under CNCF)
Home Page: https://v6d.io
License: Apache License 2.0
This project forked from v6d-io/v6d
vineyard (v6d): an in-memory immutable data manager. (Project under CNCF)
Home Page: https://v6d.io
License: Apache License 2.0
上一步已经实现了通过LRU来追踪冷数据,这一步中我将给出实现Spill的设计方案。
为了实现能将冷数据dump到盘上的功能,我们需要
0: 探测到OOM
1: 选出冷数据, 这一步由LRU提供, 我们预期需要将多个数据块spill到盘上以达到设定的阈值
2: 将冷数据Dump到盘上
3: 设计一个spilled 数据块的取回callback
我们需要涉及的新数据结构可能有
spilling_threshold
memory usage正常的阈值spilled_obj
, 记录了spilled数据的concurrent_hash_mapspilled_path
, 记录了spill数据的path针对上述的4个步骤,分别阐述
这个不难,当我们调用 BulkStore::Create
的时候,会去调用AllocateMemory如果返回了nullptr就是OOM了
考虑到整体的上锁、解锁的开销,考虑在LRU内部提供一个新的方法 vector<Obj> PopObject(uint64_t size)
根据传入的size来返回多个Obj. (Note: 这里需要考虑到多线程的安全,比如有一个读请求进来,会不会在这个Spill的过程即没有在LRU中找到又没有在spilled_obj
中找到)
这里比较麻烦的一点是,我们是一个文件一个spilled还是一个文件内部把所有的spilled都给存了,考虑survey一下别的数据库的实现
直接可以做到spilled_obj里面,可以考虑新增一个类,封装一下对应的fd、callback
TODO:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.