Code Monkey home page Code Monkey logo

jingchengli / swapdb Goto Github PK

View Code? Open in Web Editor NEW
396.0 36.0 71.0 24.56 MB

https://github.com/JingchengLi/swapdb/wiki

License: BSD 2-Clause "Simplified" License

CMake 0.32% Makefile 0.81% C 41.09% C++ 30.18% Shell 0.73% Perl 1.25% M4 0.58% XSLT 0.09% HTML 2.91% Roff 0.21% CSS 0.02% Batchfile 0.01% Lua 0.08% Ruby 0.54% Tcl 11.82% Python 8.85% Smarty 0.01% COBOL 0.26% GAP 0.11% PHP 0.13%
redis redis-compatiable-storage capacity-redis capacity-kv-storage

swapdb's Introduction

SwapDB

A Redis-compatible storage solution that enables data exchange between memory and disk, allowing for significant savings in memory costs compared to Redis. The core concept behind SwapDB is to store frequently accessed "hot" keys in Redis memory, while infrequently accessed "cold" keys are stored on disk. As keys transition from cold to hot, they are loaded into Redis memory, and when keys transition from hot to cold, they are transferred back to disk. By leveraging the capabilities of SwapDB, users can achieve both high-performance caching and high-capacity key-value storage.

Documentation

中文文档

Fundamental

fundamental

Features

  • Heat statistics of keys(LFU)
  • Configurable threshold of RAM/FLASH capacity
  • Redis API compatiable(99%). supports data structures such as strings, hashes, lists, sets, sorted sets
  • Cluster management(redis cluster)
  • Multiple replica nodes and data replication support(RDB+Snapshot)
  • Data persistency support
  • High performance and high capacity redis-like storage

Compile

requirements:

CMake >= 3.1
GCC >= 4.8

Get the source code

git clone https://github.com/JRHZRD/swapdb.git --recursive

Let's build

(you can skip this step if you add '--recursive' option when 'git clone'.) for submodules update process.

git submodule update --init --recursive
cmake . && make -j8

Quick start

you can quickly start a swap-redis and swap-ssdb instance like this:

cd utils
# this will use the default "6379" port for swap-redis and "26379" port for swap-ssdb.
./deploy_redis.sh
# or you can specify a specific port like this, for example, use "6380" port
# ./deploy_redis.sh 6380

redis-cli -p 6379
127.0.0.1:6379> set a b
OK
127.0.0.1:6379> locatekey a
"redis"
127.0.0.1:6379> storetossdb a
OK
127.0.0.1:6379> locatekey a
"ssdb"
127.0.0.1:6379> get a
"b"
127.0.0.1:6379> dumpfromssdb a
OK
127.0.0.1:6379> locatekey a
"redis"

Applicable scenes

  • cache

SwapDB support LFU based heat statistics, hot keys are kept in redis,so you can use SwapDB as cache, which has the same performance as redis when access hot keys.

  • High capacity redis-like KV storage

By configuring a low threshold of RAM/FLASH capacity, most of the data will be stored in disk and only the hottest data stored in redis.

swapdb's People

Contributors

amallia avatar andyli029 avatar antirez avatar badboy avatar cbgbt avatar concreted avatar dvirsky avatar githubmota avatar guybe7 avatar itamarhaber avatar jingchengli avatar jrhzrd avatar jstncarvalho avatar lorneli avatar luoming1224 avatar miterv1 avatar oranagra avatar peter-xu-zhou avatar quchen88 avatar sgn1 avatar sitano avatar spinlock avatar timothyzhang023 avatar viennadd avatar whatacold avatar yossigo avatar zhukovalexander avatar zintrepid avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

swapdb's Issues

对增加数据中心/机房配置项的探讨

不好意思,我用中文表达了啊,不同意见互相交流下。
问题描述:redis在集群工作时,一个从节点发现自己的主节点进入下线状态时将进行故障转移。故障转移即是发生故障主节点的所有从节点进行选主(Raft算法领头选举)。您是觉得对于部署在和主节点不同机房的从节点被选中后跨机房通信会对性能有影响,所以增加了datacenter-id这个配置项。
我的理解:我觉得这个问题只是一个使用过程中的具体问题,如果用数据中心/机房做配置项,再来一个需求:一个机房内部不同交换机/子网之间有没有必要再增加配置项(这个例子可能不恰当,毕竟机房内部延迟足够小),我的意思是redis作为一个软件产品需要抽象具体需求来给出解决方案,所以代码层面有数据中心(机房)类似的代码是有些狭义了
延伸:其实我觉得redis cluster对于用户来说最大的问题是运维问题,过于“自动化”,对人工干预不友好,这也是好多用户不敢用的原因。我觉得在所有从节点进行Raft算法选主前应该先判断一下有没有被人工干预指定为主节点的配置项(配置信息所有从节点信息是经过同步一致的),如果有配置信息,则此从节点单独参选主节点,所有从节点都没有则大家一起参选。
用途:1.解决跨机房节点被选主带来性能问题
2.人为干预,可以平移整个集群到另一个机房(实现集群的平移)
3.使得不同机房的从(备)节点更加有实际意义

在centos7上编译失败

/data/swapdb/swap-redis-4.0.0/deps/linenoise/linenoise.c: 在函数‘refreshMultiLine’中:
/data/swapdb/swap-redis-4.0.0/deps/linenoise/linenoise.c:568:27: 警告:ISO C99 需要使用剩余的参数 [默认启用]
lndebug("clear+up");
^
/data/swapdb/swap-redis-4.0.0/deps/linenoise/linenoise.c:574:20: 警告:ISO C99 需要使用剩余的参数 [默认启用]
lndebug("clear");
^
/data/swapdb/swap-redis-4.0.0/deps/linenoise/linenoise.c:591:28: 警告:ISO C99 需要使用剩余的参数 [默认启用]
lndebug("");
^
/data/swapdb/swap-redis-4.0.0/deps/linenoise/linenoise.c:619:17: 警告:ISO C99 需要使用剩余的参数 [默认启用]
lndebug("\n");
^
[ 16%] Building C object build/SWAP_REDIS/CMakeFiles/redis-check-rdb.dir/src/blocked.c.o
/data/swapdb/swap-redis-4.0.0/src/bitops.c: 在函数‘bitfieldCommand’中:
/data/swapdb/swap-redis-4.0.0/src/bitops.c:213:54: 警告:此函数中的‘wrapped’在使用前可能未初始化 [-Wmaybe-uninitialized]
bitval = (value & ((uint64_t)1<<(bits-1-j))) != 0;
^
/data/swapdb/swap-redis-4.0.0/src/bitops.c:1053:42: 附注:‘wrapped’在此声明
uint64_t oldval, newval, wrapped, retval;
^
[ 17%] Building C object build/SWAP_REDIS/CMakeFiles/redis-check-rdb.dir/src/childinfo.c.o
[ 18%] Building C object build/SWAP_REDIS/CMakeFiles/redis-server.dir/src/blocked.c.o
[ 18%] Building C object build/SWAP_REDIS/CMakeFiles/redis-server.dir/src/childinfo.c.o
/data/swapdb/swap-redis-4.0.0/src/bitops.c: 在函数‘bitfieldCommand’中:
/data/swapdb/swap-redis-4.0.0/src/bitops.c:213:54: 警告:此函数中的‘wrapped’在使用前可能未初始化 [-Wmaybe-uninitialized]
bitval = (value & ((uint64_t)1<<(bits-1-j))) != 0;
^
/data/swapdb/swap-redis-4.0.0/src/bitops.c:1053:42: 附注:‘wrapped’在此声明
uint64_t oldval, newval, wrapped, retval;
^
[ 18%] Building C object build/SWAP_REDIS/CMakeFiles/redis-check-rdb.dir/src/cluster_v3.c.o

最终编译结束时报错

[100%] Linking CXX executable ../ssdb-server
CMakeFiles/ssdb-server.dir/src/serv.cpp.o:在函数‘proc_debug(Context&, Link*, std::vector<Bytes, std::allocator > const&, Response*)’中:
/data/swapdb/swap-ssdb-1.9.2/src/serv.cpp:586:对‘rocksdb::WriteBatch::WriteBatch(unsigned long, unsigned long)’未定义的引用
../lib/libssdb.a(ssdb_impl.cpp.o):在函数‘SSDBImpl::resetRepopid(Context&)’中:
/data/swapdb/swap-ssdb-1.9.2/src/ssdb/ssdb_impl.cpp:215:对‘rocksdb::WriteBatch::WriteBatch(unsigned long, unsigned long)’未定义的引用
../lib/libssdb.a(ssdb_impl.cpp.o):在函数‘SSDBImpl::flushdb(Context&)’中:
/data/swapdb/swap-ssdb-1.9.2/src/ssdb/ssdb_impl.cpp:288:对‘rocksdb::WriteBatch::WriteBatch(unsigned long, unsigned long)’未定义的引用
/data/swapdb/swap-ssdb-1.9.2/src/ssdb/ssdb_impl.cpp:319:对‘rocksdb::WriteBatch::WriteBatch(unsigned long, unsigned long)’未定义的引用
../lib/libssdb.a(ssdb_impl.cpp.o):在函数‘SSDBImpl::delete_key_loop(std::string const&)’中:
/data/swapdb/swap-ssdb-1.9.2/src/ssdb/ssdb_impl.cpp:646:对‘rocksdb::WriteBatch::WriteBatch(unsigned long, unsigned long)’未定义的引用
../lib/libssdb.a(t_zset.cpp.o):/data/swapdb/swap-ssdb-1.9.2/src/ssdb/t_zset.cpp:64: more undefined references to `rocksdb::WriteBatch::WriteBatch(unsigned long, unsigned long)' follow
collect2: 错误:ld 返回 1
make[2]: *** [build/ssdb-server] 错误 1
make[1]: *** [build/SWAP_SSDB/CMakeFiles/ssdb-server.dir/all] 错误 2
make: *** [all] 错误 2

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.