Add an overview doc in both English/Chinese to explain dragonboat basics.

Add the overview doc link to README.md and also mention it in the example repo.

To support disk based state machine, snapshot image format need to be upgraded to

• support block based checksum.
• allow the total size to be unknown when start streaming.
• support metadata only snapshot.

The SnapshotWriter and SnapshotReader types should be extended to support the above described new features and maintain backwards capability to support the current snapshot format.

The coming new release will have the version number bumped to v3.0. It is a major release with the following breaking changes -

• NewNodeHost now returns error when it fails to create a new NodeHost instance.

Your existing code is probably doing this -

nh := dragonboat.NewNodeHost(nhc)

Please update it to

nh, err := dragonboat.NewNodeHost(nhc)
if err != nil {
  panic(err) // or other err handling you want
}

• In v3.0, the Update method in IStateMachine will be updated to return a Result instance and an error, it contains both an uint64 integer value and a byte slice. This is based on user request to support CompareAndSwap style operation.

Existing user applications can be easily updated to use the new API. For existing Update() method:

return v

should be changed to:

return statemachine.Result{Value: v}, nil 

• in v3.0, the Lookup method's signature will be changed to be Lookup(interface{}) (interface{}, error)

you can also choose to implement the optional NALookup([]byte) ([]byte, error) method to retain the old interface and behaviour.

• in v3.0, the GetHash method will be changed to return (uint64, error), it also becomes an optional method, you no longer have to implement it.

• In v3.0, the SaveSnapshot no longer require the number of written bytes to be returned. This is based on observation that multiple users got confused by the existing API.

Existing user applications can be easily updated to use the new API. For existing SaveSnapshot() method:

return numOfWrittenBytes, nil

should be changed to

return nil

Note that the v2.x release will continue to be supported for fixing critical bugs. Users are free to choose whether to the use v2.x which is more stable or v3.0 for more advanced features.

• Drummer has been moved into the internal package. The MasterClient interface has been replaced by a GetNodeHostInfo() API.

• The C++ binding will be removed from the dragonboat repo. It might be moved to a separate repo in the future if someone can offer to maintain it.

New Feature

• Item 1
• Item 2
• Item 3
• Item 4

new features are separate from the real problem list, which is better.

how to dynamically add node?

Dragonboat allows users to use ReadIndex() + ReadLocal() to performance linearizable reads.

I couldn't think of any legitimate reason for restarting a node in between, the timing also makes it highly unlikely to happen, but it is possible in theory using the current API.

I haven't seen such issue in production/test/dev, haven't heard anything from anyone, but still need to update nodehost.go to make it impossible to involve a restarted node when ReadIndex() + ReadLocal() are used. Just make it harder to abuse the API.

If I have a lot of data, I need to take a snapshot, It might take two seconds.
during these two seconds, the state machine cannot update?

Add API to request a new snapshot to be created, its usage should be similar to the one used for requesting raft membership change.

i am reading the source code, and found the follower simple accept log from leader and append to slice(not to DB or say Persistence). that's not what RAFT protocol said.
am i read code wrong or something else? thanks for you time.

Non-voting observer member support has been recently added, but such observer status info is not available to the IMasterClient and thus not available to drummer.

This is a low priority issue as both non-voting member/drummer features are optional.

github.com/snlansky/silk/vendor/github.com/lni/dragonboat/internal/cpp

In file included from ../vendor/github.com/lni/dragonboat/internal/cpp/wrapper.go:34:0:
./wrapper.h:19:10: fatal error: dragonboat/binding.h: No such file or directory
#include "dragonboat/binding.h"
^~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
When I use dep or go mod, the header file will not be found.This makes it impossible to compile the go binary using the docker image.

Inspired by #52 , wondering whether it makes sense to trigger jenkins test coverage for major codepaths in dragonboat.

do you have performance tunning document?
or some parameters?
@lni

type IStateMachine interface {
  Update(data []byte)  uint64
}

in a real world, we cann't assume Update() always have the same result. If one of node failed to Update, lastApplied is still updated. How can we avoid this ?

When a node fails, how to get the it’s state ?

When using the optional Master server, NodeHost periodically sends its details to the Master server. There is only one RaftAddress right now. We use it both for binding address and advertise address. there are cases we cannot use the RaftAddress as the binding address, e.g. when private/public address mapping is involved.

Is there any chance to add an advertise address?
if advertise address is not set, nothing changed.
if advertise address is set, use advertise address for identification.

2019-03-20 14:19:45.212475 I | dragonboat: [00017:00003] snapshotted, index 101, term 2, file count 2
2019-03-20 14:19:45.236230 C | raftpb: file tmp/node/127-0-0-1-2224/boy-OptiPlex-3050/00000000000000000001/snapshot-part-17/snapshot-17-3/snapshot-0000000000000065/snapshot-0000000000000065.gbsnap size 1204, expect 1040
2019-03-20 14:19:45.236273 E | utils: worker is panicing, file tmp/node/127-0-0-1-2224/boy-OptiPlex-3050/00000000000000000001/snapshot-part-17/snapshot-17-3/snapshot-0000000000000065/snapshot-0000000000000065.gbsnap size 1204, expect 1040
panic: file tmp/node/127-0-0-1-2224/boy-OptiPlex-3050/00000000000000000001/snapshot-part-17/snapshot-17-3/snapshot-0000000000000065/snapshot-0000000000000065.gbsnap size 1204, expect 1040 [recovered]
	panic: file tmp/node/127-0-0-1-2224/boy-OptiPlex-3050/00000000000000000001/snapshot-part-17/snapshot-17-3/snapshot-0000000000000065/snapshot-0000000000000065.gbsnap size 1204, expect 1040

goroutine 109 [running]:
github.com/lni/dragonboat/internal/utils/syncutil.(*Stopper).runWorker.func1.1()
	/home/boy/go/pkg/mod/github.com/lni/[email protected]/internal/utils/syncutil/stopper.go:84 +0xeb
panic(0xb183c0, 0xc04ed445a0)
	/usr/local/go/src/runtime/panic.go:522 +0x1b5
github.com/lni/dragonboat/internal/utils/logutil/capnslog.(*PackageLogger).Panicf(0xc04edef0a0, 0xc4ad79, 0x1a, 0xc002d8ff20, 0x3, 0x3)
	/home/boy/go/pkg/mod/github.com/lni/[email protected]/internal/utils/logutil/capnslog/pkg_logger.go:83 +0x135
github.com/lni/dragonboat/logger.(*capnsLog).Panicf(0xc0000be668, 0xc4ad79, 0x1a, 0xc002d8ff20, 0x3, 0x3)
	/home/boy/go/pkg/mod/github.com/lni/[email protected]/logger/capnslogger.go:73 +0x60
github.com/lni/dragonboat/logger.(*dragonboatLogger).Panicf(0xc00017b890, 0xc4ad79, 0x1a, 0xc002d8ff20, 0x3, 0x3)
	/home/boy/go/pkg/mod/github.com/lni/[email protected]/logger/logger.go:120 +0x74
github.com/lni/dragonboat/raftpb.checkFileSize(0xc04f1f0460, 0x99, 0x410)
	/home/boy/go/pkg/mod/github.com/lni/[email protected]/raftpb/raft.go:242 +0x1f5
github.com/lni/dragonboat/raftpb.(*Snapshot).Validate(0xc04cb48b40, 0xc04f1f0460)
	/home/boy/go/pkg/mod/github.com/lni/[email protected]/raftpb/raft.go:253 +0x56
github.com/lni/dragonboat.(*node).saveSnapshot(0xc000124700)
	/home/boy/go/pkg/mod/github.com/lni/[email protected]/node.go:434 +0x95c
github.com/lni/dragonboat.(*execEngine).saveSnapshot(0xc00026c460, 0x11, 0xc000208f00)
	/home/boy/go/pkg/mod/github.com/lni/[email protected]/execengine.go:241 +0x206
github.com/lni/dragonboat.(*execEngine).snapshotWorkerMain(0xc00026c460, 0x12)
	/home/boy/go/pkg/mod/github.com/lni/[email protected]/execengine.go:216 +0x4af
github.com/lni/dragonboat.newExecEngine.func3()
	/home/boy/go/pkg/mod/github.com/lni/[email protected]/execengine.go:140 +0x33
github.com/lni/dragonboat/internal/utils/syncutil.(*Stopper).runWorker.func1(0xc0000b1e40, 0xc0000ba4d0, 0x0, 0x0, 0xc00024a500)
	/home/boy/go/pkg/mod/github.com/lni/[email protected]/internal/utils/syncutil/stopper.go:91 +0x5d
created by github.com/lni/dragonboat/internal/utils/syncutil.(*Stopper).runWorker
	/home/boy/go/pkg/mod/github.com/lni/[email protected]/internal/utils/syncutil/stopper.go:80 +0x98

I got panic when save sanpshot. using version v2.1.3.
I'm using multiplile goroutine to wirte

sess := n.nh.GetNoOPSession(clusterId)
v, err = n.nh.SyncPropose(cctx, sess, b)

It is currently used in drummer tests and by the binding package. In a normal build, try not to import the internal/cpp package.

Once internal/cpp is avoided in a normal build, together with the EXPERIMENTAL pebble support, we should be able to build dragonboat as pure Go project.

Hi,

I have a few subscribers that needs to do syncReads once new data for that specific topic or item has changed or is added. Is there a way to accomplish this currently? Currently it looks like the only way is to poll or long poll through the read interface. It would be nice if I can register some hooks or have an update channel.

Thanks!

Dockerfile
`
FROM golang:1.11
MAINTAINER silk
ENV GOCACHE=off
ENV GOPATH=/opt/gopath
ENV CGO_CFLAGS="-I/path/to/rocksdb/include"
ENV CGO_LDFLAGS="-L/path/to/rocksdb/lib -lrocksdb"

RUN mkdir -p "$GOPATH/src" "$GOPATH/bin" && chmod -R 777 "$GOPATH"
WORKDIR $GOPATH

RUN apt-get update && apt-get install -y build-essential libncurses-dev libtool make gcc g++ unzip

RUN go get -u -d github.com/lni/dragonboat;
cd $GOPATH/src/github.com/lni/dragonboat;
make install-rocksdb-ull
`
ln -fs librocksdb.so.5.13.4 librocksdb.so
ln -fs librocksdb.so.5.13.4 librocksdb.so.5
ln -fs librocksdb.so.5.13.4 librocksdb.so.5.13
make[1]: Leaving directory '/opt/gopath/src/github.com/lni/dragonboat/build/rocksdbtmp/rocksdb-5.13.4'
Makefile:265: recipe for target 'install-rocksdb-lib-ull' failed
/bin/sh: 3: sudo: not found
make: *** [install-rocksdb-lib-ull] Error 127

What are the advantages， thanks.

and confusing.

need at least complete working examples for several nodes

Unable to PASS the tests for the C++ bindings installed. Here's the console output:

rocksdb lib found at /usr/local/lib/librocksdb.5.dylib
TESTBUILD flag set, doing a DEBUG build
build tags are set to " "
CGO_LDFLAGS="-L/usr/local/lib -lrocksdb"  go build -tags="  dragonboat_language_binding" -o binding/libdragonboat.so -buildmode=c-shared \
		github.com/lni/dragonboat/binding
if [ Darwin = "Darwin" ] ; \
  then \
    install_name_tool -id "@rpath/libdragonboat.so" binding/libdragonboat.so; \
  fi
ln -f -s /Users/USERNAME/go/src/github.com/lni/dragonboat/binding/libdragonboat.h \
		binding/include/dragonboat/libdragonboat.h
mv binding/libdragonboat.so libdragonboat.so.2.0.0
ln -fs libdragonboat.so.2.0.0 libdragonboat.so
ln -fs libdragonboat.so.2.0.0 libdragonboat.so.2
ln -fs libdragonboat.so.2.0.0 libdragonboat.so.2.0
c++ -std=c++11 -Wall -fPIC -O3 -Ibinding/include -g -fsanitize=address -fsanitize=undefined -c -o binding/cpp/dragonboat.o binding/cpp/dragonboat.cpp
cc -Wall -fPIC -O3 -Ibinding/include -g -fsanitize=address -fsanitize=undefined -c -o binding/c/binding.o binding/c/binding.c
ar rcs libdragonboatcpp.a binding/cpp/dragonboat.o binding/c/binding.o
c++ -std=c++11 -Wall -fPIC -O3 -Ibinding/include -g -fsanitize=address -fsanitize=undefined -Iinternal/tests -Ibinding/include -c -o internal/tests/cpptest/example.o internal/tests/cpptest/example.cpp
c++ -std=c++11 -Wall -fPIC -O3 -Ibinding/include -g -fsanitize=address -fsanitize=undefined -Iinternal/tests -Ibinding/include -c -o internal/tests/cpptest/exampleplugin.o internal/tests/cpptest/exampleplugin.cpp
c++ -fsanitize=address -fsanitize=undefined -bundle -undefined dynamic_lookup -Wl,-install_name,dragonboat-cpp-plugin-example.so \
		-o dragonboat-cpp-plugin-example.so internal/tests/cpptest/example.o internal/tests/cpptest/exampleplugin.o
c++ -std=c++11 -Wall -fPIC -O3 -Ibinding/include -g -fsanitize=address -fsanitize=undefined -c -o binding/tests/nodehost_tests.o binding/tests/nodehost_tests.cpp
binding/tests/nodehost_tests.cpp:23:10: fatal error: 'gtest/gtest.h' file not found
#include "gtest/gtest.h"
         ^~~~~~~~~~~~~~~
1 error generated.
make: *** [binding/tests/nodehost_tests.o] Error 1

Is there something I am missing or should the header be explicitly sourced? Please advise.

TIA!

Stuck in rocksdb installing when make install-rocksdb-ull,anyone help?

gzip: stdin: unexpected end of file
rocksdb-5.13.4/monitoring/instrumented_mutex.cc
rocksdb-5.13.4/monitoring/instrumented_mutex.h
rocksdb-5.13.4/monitoring/iostats_context.cc
rocksdb-5.13.4/monitoring/iostats_context_imp.h
rocksdb-5.13.4/monitoring/iostats_context_test.cc
rocksdb-5.13.4/monitoring/perf_context.cc
rocksdb-5.13.4/monitoring/perf_context_imp.h
rocksdb-5.13.4/monitoring/perf_level.cc
rocksdb-5.13.4/monitoring/perf_level_imp.h
rocksdb-5.13.4/monitoring/perf_step_timer.h
rocksdb-5.13.4/monitoring/statistics.cc
rocksdb-5.13.4/monitoring/statistics.h
rocksdb-5.13.4/monitoring/statistics_test.cc
rocksdb-5.13.4/monitoring/thread_status_impl.cc
rocksdb-5.13.4/monitoring/thread_status_updater.cc
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now
make: *** [get-rocksdb] Error 2

Originally posted by @funlake in #4 (comment)

您好，dragonboat是个挺棒的multiraft库，挺感谢开源的.

在使用介绍里，它需要依赖rocksdb、cgo，我有点忌惮对cgo的依赖(主要是对cgo了解的不深入，网上有了解到它的一些问题)，我们在基于badger(https://github.com/dgraph-io/badger，纯go的kv存储)实现一些存储，dragonboat是否可以不用依赖rocksdb、cgo?

When running make install-rocksdb-ull, there is an error:

In file included from memtable/skiplistrep.cc:6:
./memtable/inlineskiplist.h: In instantiation of 'void rocksdb::InlineSkipList<Comparator>::Node::StashHeight(int) [with Comparator = const rocksdb::MemTableRep::KeyComparator&]':
./memtable/inlineskiplist.h:612:3:   required from 'rocksdb::InlineSkipList<Comparator>::Node* rocksdb::InlineSkipList<Comparator>::AllocateNode(size_t, int) [with Comparator = const rocksdb::MemTableRep::KeyComparator&; size_t = long unsigned int]'
./memtable/inlineskiplist.h:574:13:   required from 'rocksdb::InlineSkipList<Comparator>::InlineSkipList(Comparator, rocksdb::Allocator*, int32_t, int32_t) [with Comparator = const rocksdb::MemTableRep::KeyComparator&; int32_t = int]'
memtable/skiplistrep.cc:28:28:   required from here
./memtable/inlineskiplist.h:283:11: error: 'void* memcpy(void*, const void*, size_t)' writing to an object of type 'struct std::atomic<rocksdb::InlineSkipList<const rocksdb::MemTableRep::KeyComparator&>::Node*>' with no trivial copy-assignment [-Werror=class-memaccess]
     memcpy(&next_[0], &height, sizeof(int));
     ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from ./memtable/inlineskiplist.h:47,
                 from memtable/skiplistrep.cc:6:
/usr/include/c++/8.2.1/atomic:352:12: note: 'struct std::atomic<rocksdb::InlineSkipList<const rocksdb::MemTableRep::KeyComparator&>::Node*>' declared here
     struct atomic<_Tp*>
            ^~~~~~~~~~~~

gcc version 8.2.1 20181127 (GCC)

This is a known issue in RocksDB: facebook/rocksdb#2705

After discussions in #4, I've add some experimental code to make badger to work with Dragonboat. The code is available in the following branch -

https://github.com/lni/dragonboat/tree/badger-logdb-experimental

As mentioned above, this is some experimental code use to see how badger works. It should never be used in production.

I'd like to point out that there are outstanding issues preventing badger support to be added -

• it leaks goroutines (dgraph-io/badger#685)
• sstables are not checksumed (dgraph-io/badger#680)

There are probably more, will update when identified.

https://github.com/tidwall/buntdb

https://github.com/xujiajun/nutsdb

we config SnapshotEntries 10000, and a node restarted, 5 minites later it restarted and just 11000 late from master. the node still needs a full snapshot to recover. let's say the full snapshot is about 10m entries which is too big.
Is there any way to notify master to generate a smaller snapshot by telling it some extra info ?

https://github.com/lni/dragonboat/blob/372182542c82c2e18a3e32763081856f6a223dfb/internal/rsm/statemachine.go#

strings.ToLower(strings.TrimSpace(string(addr2)))

should be:

strings.ToLower(strings.TrimSpace(addr2))

The 2nd example “State Machine” under dragonboat-example takes us to https://github.com/lni/dragonboat-example/blob/master/helloworld/README.DS.md

Can we have more instructions on the consumption or execution of the statemachine.go file?

The statemachine.go file exists here but I think we can prescribe more instructions of consuming this in a customised manner for a test example.

Happy to understand more on using this file & contribute to the docs. Cheers.

do we have any plan to add java bindings support?
or if I want to use dragonboat in java project, do we have any suggested methods?

after passing all tests in dragonboat-cppwrapper-tests:

==1027==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 29568 byte(s) in 528 object(s) allocated from:
    #0 0x7f14dcc34458 in operator new(unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xe0458)
    #1 0x7f14dc38cba1 in leveldb_filterpolicy_create_bloom /home/jccai/go/src/github.com/lni/dragonboat/internal/logdb/levigo/deps_leveldb_db_c.cc:523

Indirect leak of 12672 byte(s) in 528 object(s) allocated from:
    #0 0x7f14dcc34458 in operator new(unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xe0458)
    #1 0x7f14dc3bd41c in leveldb::NewBloomFilterPolicy(int) /home/jccai/go/src/github.com/lni/dragonboat/internal/logdb/levigo/deps_leveldb_util_bloom.cc:92

Dragonboat's drummer package depends on golang.org/x/net, golang.org/x/text and golang.org/x/sys, all of them are not accessible for users from China mainland due to well known reasons.

To work around the problem -

Solution 1 - use https proxy to bypass GFW
Assuming you have a https proxy running at hostname:port that can bypass the GFW, then

$ https_proxy=http://hostname:port go get -u -d github.com/lni/dragonboat

Solution 2 - get golang.org/x packages from their mirrors on github.com

$ mkdir -p $GOPATH/src/golang.org/x/
$ cd $GOPATH/src/golang.org/x/
$ git clone https://github.com/golang/net.git net
$ git clone https://github.com/golang/text.git text
$ git clone https://github.com/golang/sys.git sys
$ go install net
$ go install text
$ go install sys
$ go get -d github.com/lni/dragonboat

Shortly before the dragonboat source code was made publicly available in Jan. 2019, some changes were made to include features not directly related to consensus. For example, the C++ binding was included in the release and the Master server support was refactored from the Drummer package hoping users might be able to use it.

It has become clear to me now that dragonboat should just focus on the core consensus features. In the next few weeks -

1. the IMasterClient interface in master.go will be removed. A method named GetNodeHostInfo() will be added to the NodeHost struct, it returns the current NodeHostInfo instance. Users can choose how their Master client/server are going to be built, other than providing the above NodeHostInfo record, Dragonboat will no longer get involved in how distributed NodeHost instances are managed.

2. The drummer package will be updated accordingly and it will be moved to internal/drummer. It can still be used as a reference implementation of the Master server concept but it will become an internal package.

3. The future of the C++ binding will be decided once the above item 1 and 2 are done. It might be moved to a separate repo (e.g. github.com/lni/dragonboat-cpp).

The following fails when using the C++ bindings:

status = nh->StartCluster(peers, false, "/Users/dict/git/dragonboat-test/dragonboat-cpp-plugin-cpphelloworld.so", config);

goroutine 17 [running, locked to thread]:
github.com/lni/dragonboat/internal/utils/fileutil.GetAppNameFromFilename(...)
	/Users/dict/.go/src/github.com/lni/dragonboat/internal/utils/fileutil/plugin.go:32
github.com/lni/dragonboat.(*NodeHost).StartClusterUsingPlugin(0xc0001a4000, 0xc0000cb200, 0x0, 0xc000612000, 0x47, 0x1, 0x80, 0x100, 0x5, 0x1, ...)
	/Users/dict/.go/src/github.com/lni/dragonboat/nodehost.go:394 +0x1b5
main.NodeHostStartCluster(0x2, 0x102a29b50, 0x102a282d0, 0x3, 0x0, 0x102a28d70, 0x47, 0x1, 0x80, 0x50000100, ...)
	/Users/dict/.go/src/github.com/lni/dragonboat/binding/binding.go:291 +0x395
main._cgoexpwrap_2a7f7706cd22_NodeHostStartCluster(0x2, 0x102a29b50, 0x102a282d0, 0x3, 0xa00000000, 0x102a28d70, 0x47, 0x1, 0x80, 0x50000100, ...)
	_cgo_gotypes.go:369 +0xad

It works when a relative path is used instead:

status = nh->StartCluster(peers, false, "dragonboat-cpp-plugin-cpphelloworld.so", config);

The problem seems to originate here:

A potential fix would be to remove the ^ from the beginning of the applicable regular expression(s), or change it to .*.

But it may be best if we could set our own app name, rather than forcing dynamically created libraries to have a certain path and/or name.

Thanks for your work.

When trying to build drummer/server/drummercmd using go build main.go, following exception was triggered:
drummer/client/nodehost.go:387:14: dc.nh.StartClusterUsingPlugin undefined (type *dragonboat.NodeHost has no field or method StartClusterUsingPlugin)

The root cause was due to this inclusion refactoring diff. It moved function StartClusterUsingPlugin into outside class binding.go, so drummer/client/nodehost.go couldn't access it.

This feature in RocksDB is the suggested method to delete a selected range of KVs. Sadly, with recently reported/fixed bugs in RocksDB, I just have the feeling that it is not stable enough to be used in production.

Stop using delete range+manual compaction in RocksDB based LogDB, revert back to the regular delete by key style approach. It might be slower but far better than a risky one slightly faster.

IStateMachine.Update always receives duplicate data when the node is restarted.

DragonBoatDocs is mentioned several times in the code comments, but I can't find DragonBoatDocs in the repo.

@lni said in the https://gitter.im/lni/dragonboat: "I will remove those comments. those docs are my personal notes, most of the content will be published as blog articles once it is cleaned up."

Once a node is removed (NodeHost.RequestDeleteNode) from the Raft cluster, all its data can be removed from the system to free up storage space. Add API to do that.

This is requested by a potential user.

With the new optimization in logdb, this is no longer that useful.

Once it is removed, with recent change to rocksdb's c binding, we can actually stop the rdbpatched supported.

Trigger a snapshot every 10 Update ？
why ? and how to configure it ?

from ExampleStateMachine.Update(), msg: , count:1213
2019-01-24 15:07:49.986237 I | dragonboat: snapshot at index 1221 requested on [00128:00001]
2019-01-24 15:07:49.986269 I | dragonboat: reportRequestedSnapshot, [00128:00001]
2019-01-24 15:07:49.986286 I | dragonboat: [00128:00001] called saveSnapshot
2019-01-24 15:07:50.005745 I | dragonboat: [00128:00001] snapshotted, index 1221, term 48, file count 0

2019-01-24 15:07:50.042995 I | dragonboat: [00128:00001] received completed snapshot rec {0 false false true []}
2019-01-24 15:07:50.078608 I | dragonboat: [00128:00001] compacted log at index 1216
from ExampleStateMachine.Update(), msg: , count:1214

from ExampleStateMachine.Update(), msg: , count:1215

from ExampleStateMachine.Update(), msg: , count:1216

from ExampleStateMachine.Update(), msg: , count:1217

from ExampleStateMachine.Update(), msg: , count:1218

from ExampleStateMachine.Update(), msg: , count:1219

from ExampleStateMachine.Update(), msg: , count:1220

from ExampleStateMachine.Update(), msg: , count:1221

from ExampleStateMachine.Update(), msg: , count:1222

from ExampleStateMachine.Update(), msg: , count:1223

from ExampleStateMachine.Update(), msg: , count:1224
2019-01-24 15:07:52.186224 I | dragonboat: snapshot at index 1232 requested on [00128:00001]
2019-01-24 15:07:52.186250 I | dragonboat: reportRequestedSnapshot, [00128:00001]
2019-01-24 15:07:52.186262 I | dragonboat: [00128:00001] called saveSnapshot
2019-01-24 15:07:52.215899 I | dragonboat: [00128:00001] snapshotted, index 1232, term 48, file count 0
2019-01-24 15:07:52.250311 I | dragonboat: [00128:00001] received completed snapshot rec {0 false false true []}

Cool project... Would love to learn more about the history. It looks as if it was all deleted. If you can share a branch with the old (perhaps messy) history, that might be interesting to see where you went wrong and what you learned, etc... Cheers!

When all state machine data is stored on disk, there is no need to periodically generate snapshots as the state is always available on disk.

Add such disk based state machine support to dragonboat and allow snapshot to be generated and streamed when it is actually required.

Note that this is more like an optimization to further save disk space and bandwidth consumption. The current released version of Dragonboat can easily handle State Machines that use disks heavily.

Experimental support for lmdb has been added to the branch below
https://github.com/lni/dragonboat/tree/lmdb-logdb-experimental

it is still a cgo based approach, but much easier to use/deploy -

• no extra (visible) compilation or installation step
• no need to specify CGO_LDFLAGS or CGO_CXXFLAGS
• much faster to build

it is really not that different from a pure-go approach.

RocksDB will continue to be the default Raft log storage solution, lmdb support can be enabled by setting the go build tag "dragonboat_lmdb".

There are more work pending before the lmdb support can be merged, mostly are just clean ups and testings.

master分支和release2.0都会报这个错误：

CGO_CFLAGS="-I$RAFT_HOME/include" CGO_LDFLAGS="-L$RAFT_HOME/lib -lrocksdb" go build -v 
github.com/lni/dragonboat/internal/logdb/gorocksdb
# github.com/lni/dragonboat/internal/logdb/gorocksdb
internal/logdb/gorocksdb/optionsnoop.go:5:13: undefined: Options
internal/logdb/gorocksdb/optionsnoop.go:8:13: undefined: Options
internal/logdb/gorocksdb/optionsnoop.go:11:13: undefined: Options
internal/logdb/gorocksdb/optionsnoop.go:14:13: undefined: ReadOptions
internal/logdb/gorocksdb/optionsnoop.go:17:13: undefined: CompactionOptions

echo $CGO_CFLAGS
-I/data02/wangyl11/raft/include
echo $CGO_LDFLAGS
-L/data02/wangyl11/raft/lib -lrocksdb

rocksdb version 5.18
gcc 5.2.0
go 1.10(source compiled)

Can I just compact log without snapshot , how ?
Cause, I use levelDB for kv statemachine ,not mem kv store like btree , no snapshot needed .
And ,In the case of large concurrent reads and writes, does snapshot affect system performance?