A fun project for evaluating some new optimizations quickly, do not use it in production
License: Apache License 2.0
A fun project for evaluating some new optimizations quickly, do not use it in production.
make
Put the binary of pd-server, node and tidb-server into a single dir.
Under the binary dir, run the following commands:
mkdir -p data
./pd-server
./node --db-path=data
./tidb-server --store=tikv --path="127.0.0.1:2379"
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
raftstore coprocessor is too complex, needs to be replaced by a simpler mechanism.
During benchmark tests, the unistore server may be restarted serveral time, and the next restart may fail and reporting error log
main.go:254: ^[[0;31m[fatal] Unable to find log file. Please retry ^[[0m
and following restart operations will all fail like this
sysbench oltp_read_write test tests
txn 414441650695176250 commit secondary keys failed, and report error like
14:10:31.708 +08:00] [ERROR] [2pc.go:932] ["2PC failed commit key after primary key committed"] [error="[kv:8022]Error: KV error safe to retry retryable: replaced by another transaction [try again later]"] [txnStartTS=414441650695176250] [commitTS=414441650695176310]
above this tidb log another txn 414441650695176272 blocked by 414441650695176250 once pessimistic retry on this key, and this retry seems success and leads to 414441650695176250 commit error(lock has been replaced by another transaction)
14:10:31.706 +08:00] [INFO] [adapter.go:612] ["pessimistic write conflict, retry statement"] [conn=4] [txn=414441650695176272] [forUpdateTS=414441650695176272] [err="[kv:9007]Write conflict, txnStartTS=414441650695176272, conflictStartTS=414441650695176250, conflictCommitTS=414441650695176310, key={tableID=48, handle=1} primary=[]byte(nil) [try again later]"]
unistore related log
error log 414441650695176272 commit failed on this key
14:10:31.686217 mvcc.go:663: ^[[0;31m[error] key=[116 128 0 0 0 0 0 0 48 95 114 128 0 0 0 0 0 0 1] startTS=414441650695176250 not equal lock.StartTS= 414441650695176272 commitTS=414441650695176310 lock={{414441650695176272 4144416 50695176310 0 20014 5 false 19} [116 128 0 0 0 0 0 0 54 95 114 128 0 0 0 0 0 2 1 56] [] [] []}^[[0m
414441650695176272 has build pessimistic lock(which should report error)
14:10:31.685648 mvcc.go:250: ^[[0;37m[info] [for debug] startTS=41444 1650695176272 build pessimistic lock key=[116 128 0 0 0 0 0 0 48 95 114 128 0 0 0 0 0 0 1] lock=&{{414441650695176272 414441650695176310 0 20014 5 false 19} [11 6 128 0 0 0 0 0 0 54 95 114 128 0 0 0 0 0 2 156] [] [] []}^[[0m
441650695176250 build prewrite lock before 414441650695176272
14:10:31.680253 mvcc.go:530: ^[[0;37m[info] [for debug] buildPrewrite Lock succ key=[116 128 0 0 0 0 0 0 48 95 114 128 0 0 0 0 0 0 1] Op=0 startTS=414 441650695176250 lock=&{{414441650695176250 414441650695176266 414441650695176267
441650695176250 build pessimistic lock
14:10:31.677682 mvcc.go:250: ^[[0;37m[info] [for debug] startTS=41444 1650695176250 build pessimistic lock key=[116 128 0 0 0 0 0 0 48 95 114 128 0 0 0 0 0 0 1] lock=&{{414441650695176250 414441650695176266 0 20015 5 false 19} [11 6 128 0 0 0 0 0 0 53 95 114 128 0 0 0 0 0 147 13] [] [] []}^[[0m
func (store *MVCCStore) PessimisticLock(reqCtx *requestCtx, req *kvrpcpb.PessimisticLockRequest) (*lockwaiter.Waiter, error) {
mutations := sortMutations(req.Mutations)
startTS := req.StartVersion
regCtx := reqCtx.regCtx
hashVals := mutationsToHashVals(mutations)
regCtx.AcquireLatches(hashVals)
defer regCtx.ReleaseLatches(hashVals)
batch := store.dbWriter.NewWriteBatch(startTS, 0, reqCtx.rpcCtx)
for _, m := range mutations {
// seems the pessimistic retry here, checkConflictInLockStore did not return the
// prewrite lock by 441650695176250, and nil, nil is returnd
lock, err := store.checkConflictInLockStore(reqCtx, m, startTS)
if err == ErrAlreadyRollback {
return nil, err
}
if err != nil {
return store.handleCheckPessimisticErr(startTS, err, req.IsFirstLock, req.WaitTimeout)
}
if lock != nil {
if lock.Op != uint8(kvrpcpb.Op_PessimisticLock) {
return nil, errors.New("lock type not match")
}
if lock.ForUpdateTS >= req.ForUpdateTs {
// It's a duplicate command, we can return directly.
return nil, nil
}
// Single statement rollback key, we can overwrite it.
}
}
Currently, rollback GC only works for stand-alone mode.
We need to add a timely worker that scans RollbackStore and GC them.
We should use binary format to optimize bloom filter storage format.
(pprof) top30 -cum
Showing nodes accounting for 3.35s, 84.17% of 3.98s total
Dropped 18 nodes (cum <= 0.02s)
Showing top 30 nodes out of 68
flat flat% sum% cum cum%
0.01s 0.25% 0.25% 1.69s 42.46% github.com/dgraph-io/badger/table.(*Table).readIndex.func1
0 0% 0.25% 1.61s 40.45% runtime.mcall
0.01s 0.25% 0.5% 1.61s 40.45% runtime.schedule
0 0% 0.5% 1.59s 39.95% runtime.park_m
1.42s 35.68% 36.18% 1.42s 35.68% github.com/dgraph-io/badger/table.(*header).Decode
0.43s 10.80% 46.98% 1.40s 35.18% runtime.findrunnable
0.16s 4.02% 51.01% 0.79s 19.85% runtime.runqsteal
0 0% 51.01% 0.64s 16.08% github.com/dgraph-io/badger.Open
0 0% 51.01% 0.64s 16.08% github.com/dgraph-io/badger.newLevelsController
0 0% 51.01% 0.64s 16.08% github.com/dgraph-io/badger/table.(*Table).readIndex
0 0% 51.01% 0.64s 16.08% github.com/dgraph-io/badger/table.OpenTable
0 0% 51.01% 0.64s 16.08% main.main
0 0% 51.01% 0.64s 16.08% runtime.main
0.56s 14.07% 65.08% 0.63s 15.83% runtime.runqgrab
0 0% 65.08% 0.58s 14.57% github.com/AndreasBriese/bbloom.JSONUnmarshal
0 0% 65.08% 0.56s 14.07% encoding/json.Unmarshal
0 0% 65.08% 0.36s 9.05% encoding/json.(*decodeState).literal
0 0% 65.08% 0.36s 9.05% encoding/json.(*decodeState).object
0 0% 65.08% 0.36s 9.05% encoding/json.(*decodeState).unmarshal
0 0% 65.08% 0.36s 9.05% encoding/json.(*decodeState).value
0.36s 9.05% 74.12% 0.36s 9.05% runtime.futex
0.01s 0.25% 74.37% 0.28s 7.04% runtime.notewakeup
0 0% 74.37% 0.28s 7.04% runtime.startm
0 0% 74.37% 0.28s 7.04% runtime.wakep
0 0% 74.37% 0.27s 6.78% runtime.futexwakeup
0.11s 2.76% 77.14% 0.20s 5.03% encoding/json.checkValid
0.20s 5.03% 82.16% 0.20s 5.03% encoding/json.stateInString
0.08s 2.01% 84.17% 0.19s 4.77% encoding/json.(*decodeState).scanWhile
0 0% 84.17% 0.19s 4.77% runtime.resetspinning
0 0% 84.17% 0.17s 4.27% encoding/json.(*decodeState).literalStore
port the test code in
https://github.com/tikv/tikv/blob/master/src/coprocessor/endpoint.rs
to 'unistore/tikv' package.
Currently, region split is only triggered by internal size checker.
Need to support RegionSplit request from TiDB.
When we prewrite a LockType key, we simply remove it, the transaction status is lost.
It's OK for optimistic transactions because we never set a Lock type key as primary key.
But in pessimistic transactions, it is possible that a Lock type key is the primary key.
One solution is we change the primary key of a pessimistic transaction if it's type is Lock during the prewrite phase.
lockstore/lockstore.go:65:16: func (*node).entryLen is unused (U1000)
lockstore/lockstore.go:236:21: func (*MemStore).getNode is unused (U1000)
metrics/metrics.go:99:26: prometheus.Handler is deprecated: Please note the issues described in the doc comment of InstrumentHandler. You might want to consider using promhttp.Handler instead. (SA1019)
pd/client.go:63:2: var errFailInitClusterID is unused (U1000)
rocksdb/filter_builder.go:49:34: should use !bytes.Equal(b.lastWholeKey, key) instead (S1004)
rocksdb/filter_builder.go:74:33: should use !bytes.Equal(b.lastPrefix, prefix) instead (S1004)
rocksdb/options.go:17:2: only the first constant in this group has an explicit type (SA9004)
rocksdb/options.go:41:2: only the first constant in this group has an explicit type (SA9004)
rocksdb/sst_file_iterator.go:171:2: should use 'return ' instead of 'if { return }; return ' (S1008)
tikv/closure_exec.go:267:2: field count is unused (U1000)
tikv/closure_exec.go:268:2: field limit is unused (U1000)
tikv/closure_exec.go:661:6: this value of err is never used (SA4006)
tikv/cop_handler.go:23:5: var dummySlice is unused (U1000)
tikv/cop_handler.go:151:23: func (*evalContext).newRowDecoderForOffsets is unused (U1000)
tikv/cop_handler.go:172:23: func (*evalContext).decodeRelatedColumnVals is unused (U1000)
tikv/cop_handler_test.go:264:22: store.svr.buildDAGExecutor undefined (type *Server has no field or method buildDAGExecutor) (compile)
tikv/dbreader/db_reader.go:124:2: redundant return statement (S1023)
tikv/mvcc/tikv.go:146:3: this value of data is never used (SA4006)
tikv/raftstore/applier.go:79:6: type keyRange is unused (U1000)
tikv/raftstore/applier.go:149:2: field ranges is unused (U1000)
tikv/raftstore/applier.go:263:2: redundant return statement (S1023)
tikv/raftstore/applier.go:530:2: field merged is unused (U1000)
tikv/raftstore/applier.go:540:2: field readySourceRegion is unused (U1000)
tikv/raftstore/applier.go:1036:2: redundant return statement (S1023)
tikv/raftstore/applier.go:1094:2: redundant return statement (S1023)
tikv/raftstore/applier.go:1119:2: redundant return statement (S1023)
tikv/raftstore/applier.go:1159:2: redundant return statement (S1023)
tikv/raftstore/applier.go:1430:2: field merge is unused (U1000)
tikv/raftstore/applier.go:1431:2: field readyToMerge is unused (U1000)
tikv/raftstore/applier.go:1527:19: func (*applier).resumePendingMerge is unused (U1000)
tikv/raftstore/config.go:267:20: error strings should not end with punctuation or a newline (ST1005)
tikv/raftstore/config.go:289:20: error strings should not end with punctuation or a newline (ST1005)
tikv/raftstore/config.go:298:20: error strings should not be capitalized (ST1005)
tikv/raftstore/config.go:302:20: error strings should not end with punctuation or a newline (ST1005)
tikv/raftstore/config.go:311:20: error strings should not end with punctuation or a newline (ST1005)
tikv/raftstore/fsm_peer.go:87:20: func (*peerFsm).drop is unused (U1000)
tikv/raftstore/fsm_peer.go:91:20: func (*peerFsm).regionID is unused (U1000)
tikv/raftstore/fsm_peer.go:115:20: func (*peerFsm).hasPendingMergeApplyResult is unused (U1000)
tikv/raftstore/fsm_peer.go:232:27: func (*peerFsmDelegate).resumeHandlePendingApplyResult is unused (U1000)
tikv/raftstore/fsm_peer.go:879:27: func (*peerFsmDelegate).validateMergePeer is unused (U1000)
tikv/raftstore/fsm_peer.go:883:27: func (*peerFsmDelegate).scheduleMerge is unused (U1000)
tikv/raftstore/fsm_peer.go:887:27: func (*peerFsmDelegate).rollbackMerge is unused (U1000)
tikv/raftstore/fsm_peer.go:1198:3: should use bytes.Equal(leftKey[:tablecodec.TableSplitKeyLen], rightKey[:tablecodec.TableSplitKeyLen]) instead (S1004)
tikv/raftstore/fsm_peer.go:1409:4: should use 'return ' instead of 'if { return }; return ' (S1008)
tikv/raftstore/fsm_store.go:89:2: field cleanUpSSTScheduler is unused (U1000)
tikv/raftstore/fsm_store.go:109:24: func (*PollContext).flushLocalStats is unused (U1000)
tikv/raftstore/fsm_store.go:170:2: field lastCompactCheckKey is unused (U1000)
tikv/raftstore/fsm_store.go:171:2: field stopped is unused (U1000)
tikv/raftstore/fsm_store.go:249:2: field resolverScheduler is unused (U1000)
tikv/raftstore/fsm_store.go:250:2: field snapScheduler is unused (U1000)
tikv/raftstore/fsm_store.go:918:6: type regionIDDeclinedBytesPair is unused (U1000)
tikv/raftstore/fsm_store.go:923:6: func calcRegionDeclinedBytes is unused (U1000)
tikv/raftstore/io_limiter.go:20:2: field limiter is unused (U1000)
tikv/raftstore/msg.go:55:2: const msgDefaultChanSize is unused (U1000)
tikv/raftstore/peer.go:466:50: should use time.Since instead of time.Now().Sub (S1012)
tikv/raftstore/peer.go:1072:7: should use !bytes.Equal(state.RequestCtx, read.binaryId()) instead (S1004)
tikv/raftstore/peer.go:1085:7: should use !bytes.Equal(state.RequestCtx, read.binaryId()) instead (S1004)
tikv/raftstore/peer.go:1467:21: error strings should not end with punctuation or a newline (ST1005)
tikv/raftstore/peer.go:1489:20: error strings should not end with punctuation or a newline (ST1005)
tikv/raftstore/peer.go:1493:20: error strings should not end with punctuation or a newline (ST1005)
tikv/raftstore/peer.go:1530:23: error strings should not end with punctuation or a newline (ST1005)
tikv/raftstore/peer.go:1591:23: error strings should not end with punctuation or a newline (ST1005)
tikv/raftstore/peer.go:1705:44: error strings should not end with punctuation or a newline (ST1005)
tikv/raftstore/peer_state.go:102:2: field closeCh is unused (U1000)
tikv/raftstore/peer_storage.go:203:6: func recoverFromApplyingState is unused (U1000)
tikv/raftstore/peer_storage.go:499:22: this value of err is never used (SA4006)
tikv/raftstore/raft_client.go:122:2: should write v := c.addrs[storeID] instead of v, _ := c.addrs[storeID] (S1005)
tikv/raftstore/snap.go:140:6: func copySnapshot is unused (U1000)
tikv/raftstore/snap.go:506:10: empty branch (SA9003)
tikv/raftstore/snap_test.go:48:72: undeclared name: DBBundle (compile)
tikv/raftstore/ticker.go:100:20: using time.Tick leaks the underlying ticker, consider using it only in endless functions, tests and the main package, and use time.NewTicker here (SA1015)
tikv/raftstore/ticker.go:107:18: should omit value from range; this loop is equivalent to for regionID := range ... (S1005)
tikv/raftstore/util.go:321:20: error strings should not end with punctuation or a newline (ST1005)
tikv/raftstore/util.go:325:20: error strings should not end with punctuation or a newline (ST1005)
tikv/raftstore/worker.go:50:2: const taskTypeCompact is unused (U1000)
tikv/raftstore/worker.go:51:2: const taskTypeCheckAndCompact is unused (U1000)
tikv/raftstore/worker.go:154:6: type compactTask is unused (U1000)
tikv/raftstore/worker.go:158:6: type checkAndCompactTask is unused (U1000)
tikv/raftstore/worker.go:184:2: field closeCh is unused (U1000)
tikv/raftstore/worker.go:440:43: func (*pendingDeleteRanges).remove is unused (U1000)
tikv/raftstore/worker.go:450:43: func (*pendingDeleteRanges).timeoutRanges is unused (U1000)
tikv/raftstore/worker.go:482:24: func stalePeerInfo.timeout is unused (U1000)
tikv/raftstore/worker.go:626:74: should use time.Since instead of time.Now().Sub (S1012)
tikv/raftstore/worker.go:651:19: plainFileUsed is a pure function but its return value is ignored (SA4017)
tikv/raftstore/worker.go:876:24: func (*regionRunner).shutdown is unused (U1000)
tikv/region.go:98:22: func (*regionCtx).lessThanStartKey is unused (U1000)
tikv/region.go:106:22: func (*regionCtx).greaterThanEndKey is unused (U1000)
tikv/region.go:496:21: using time.Tick leaks the underlying ticker, consider using it only in endless functions, tests and the main package, and use time.NewTicker here (SA1015)
tikv/server.go:29:2: field wg is unused (U1000)
tikv/server.go:44:20: func (*Server).checkRequestSize is unused (U1000)
tikv/server.go:134:8: this value of err is never used (SA4006)
tikv/write.go:185:21: using time.Tick leaks the underlying ticker, consider using it only in endless functions, tests and the main package, and use time.NewTicker here (SA1015)
tikv/write.go:231:34: should use make(chan struct{}) instead (S1019)
The current badger table format doesn't have index blocks in an SST, so we need to access multiple blocks to find a key.
If we store all the base keys in index blocks, we only need to access two blocks to find a key.
This feature is required for port Snapshot related code in TiKV
https://github.com/pingcap/jepsen/blob/testing_ci/tidb/resources/tikv.conf
Those options make the test more easy to fail.
region-max-keys and region-split-keys require to implement a keySplitChecker
panic: unexpected EOF
goroutine 181 [running]:
github.com/zhangjinpeng1987/raft.(*RaftLog).Term(0xc00027f880, 0x1431, 0x1432, 0xc03d15a5e8, 0x1401)
/Users/zhou/gopath/pkg/mod/github.com/zhangjinpeng1987/[email protected]/log.go:267 +0x1e4
github.com/ngaut/unistore/tikv/raftstore.(*peerFsmDelegate).onRaftGCLogTick(0xc03d17be90)
/Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/fsm_peer.go:1155 +0x4a0
github.com/ngaut/unistore/tikv/raftstore.(*peerFsmDelegate).onTick(0xc03d17be90)
/Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/fsm_peer.go:196 +0x176
github.com/ngaut/unistore/tikv/raftstore.(*peerFsmDelegate).handleMsgs(0xc03d17be90, 0xc03d17bea0, 0x1, 0x1)
/Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/fsm_peer.go:146 +0x656
github.com/ngaut/unistore/tikv/raftstore.(*raftWorker).run(0xc0141e0d20, 0xc0000ae1e0, 0xc00014c130)
/Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/peer_state.go:153 +0x9b9
created by github.com/ngaut/unistore/tikv/raftstore.(*raftBatchSystem).startSystem
/Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/fsm_store.go:523 +0x160
Problem:
jepsen bank-multitable test on unistore, most transfer operations result in query timeout, like
WARN [jepsen worker 0-n1] jepsen.core: Process 0 crashed
java.sql.SQLTimeoutException: (conn:-1) Query timed out
...
Caused by: java.sql.SQLException: Connection timed out
...
Caused by: java.sql.SQLNonTransientConnectionException: Read timed out
From unistore server logs,
// there is a deadlock
14:38:17.652334 server.go:216: ^[[0;31m[error] deadlock found for entry={412886839456694288 412886839456694306 10001446507541019278 {} [] 0}^[[0m
2019/11/29 14:38:17.669370 mvcc.go:322: ^[[0;37m[info] 412886839496015894 blocked by 412886839456694306 on key 10001446507541019278^[[0m
// here PessimisticRollback reiceived from client
2019/11/29 14:38:17.699257 lockwaiter.go:139: ^[[0;37m[info] wakeup 3 txns blocked by txn 412886839456694288 keyHashes= [8885380622799078238] ^[[0m
// but currently unistore wont't do anything processing pessimistic lock rollback
14:38:17.522632 mvcc.go:322: ^[[0;37m[info] 412886839456694273 blocked by 412886839456694280 on key 8885380622799078238^[[0m
14:38:19.704132 mvcc.go:322: ^[[0;37m[info] 412886839456694273 blocked by 412886839456694288 on key 8885380622799078238
...
// so the woken up transaction still blocks since the lockStore contains lock info
14:38:36.741293 mvcc.go:322: ^[[0;37m[info] 412886839456694273 blocked by 412886839456694288 on key 8885380622799078238
// the pessimitic lock record in lockStore will be cleaned only when lock TTL expire, and other Transcation call like "checkTxnStatus" or "resolveLock"
So the woken up transaction blocked by pessimistic locks left by deadlock rollbacked transactions, and other transactions blocked by locks holding by this woken up transaction, thus all transactions cannot progress and most transfer operations result in "Query Timeout"
Simple step to reproduce:
create table t(c int priamry key); insert into t values(1),(2);
session 1:
select * from t where c = 1 for update;
session 2:
select * from t where c = 2 for update;
select * from t where c = 1 for update;
session 1:
select * from t where c = 2 for update;
deadlock found
but session2 still waiting on the lock, after about 20 seconds, this statement returns ok
Cleaning the rollbacked pessimistic locks on unistore-server is needed to prevent blocking other transactions and causing more blocks
related: tikv/tikv#3913
After bootstrap, the unistore cluster keep printing error logs like
raft_client.go:107: [error] raft client failed to send
transport.go:85: [error] send raft msg err. err: EOF
Need investigations.
RaftRegionManager. OnSplitRegion will add newRegionCtx into rm.regions map in another goroutine, which may lead to some event processing like OnRoleChange failing to get region from map using regionId
Maybe we need to look closely maintaining the regions map
func (rm *RaftRegionManager) OnSplitRegion(derived *metapb.Region, regions []*metapb.Region, peers []*raftstore.PeerEventContext) {
rm.mu.RLock()
oldRegion := rm.regions[derived.Id]
rm.mu.RUnlock()
// The old region is about to be replaced, update the epoch to nil to prevent accept new requests.
oldRegion.updateRegionEpoch(nil)
// Run in another goroutine to avoid deadlock.
go func() {
oldRegion.waitParent()
rm.mu.Lock()
for i, region := range regions {
rm.regions[region.Id] = newRegionCtx(region, oldRegion, peers[i].LeaderChecker)
}
rm.mu.Unlock()
oldRegion.refCount.Done()
}()
}
The raft data should be less than 10% of KV data, but its more than 70% of KV data now.
When enable fast analyze, analyzing tpcc.bmsql_stock table, tidb gorutine blocked forever on this line
1 @ 0x431170 0x404f5b 0x404f31 0x404d15 0x15e820a 0x15ec88e 0x15e6913 0x15e3946 0x45ef41
# 0x15e8209 github.com/pingcap/tidb/executor.(*AnalyzeFastExec).buildSampTask+0x679 /Users/zhou/gorepo/src/github.com/pingcap/tidb/executor/analyze.go:768
# 0x15ec88d github.com/pingcap/tidb/executor.(*AnalyzeFastExec).buildStats+0x1ad /Users/zhou/gorepo/src/github.com/pingcap/tidb/executor/analyze.go:1140
# 0x15e6912 github.com/pingcap/tidb/executor.analyzeFastExec+0x42 /Users/zhou/gorepo/src/github.com/pingcap/tidb/executor/analyze.go:539
# 0x15e3945 github.com/pingcap/tidb/executor.(*AnalyzeExec).analyzeWorker+0x375 /Users/zhou/gorepo/src/github.com/pingcap/tidb/executor/analyze.go:201
Need to investigate if it is a TiDB bug or a Unistore bug.
2019/11/19 12:30:18 Assert failed
github.com/coocood/badger/y.Assert
/Users/zhou/gopath/pkg/mod/github.com/coocood/[email protected]/y/error.go:55
github.com/ngaut/unistore/tikv/raftstore.(*applier).execCommit
/Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/applier.go:1075
github.com/ngaut/unistore/tikv/raftstore.(*applier).execWriteCmd
/Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/applier.go:883
github.com/ngaut/unistore/tikv/raftstore.(*applier).execRaftCmd
/Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/applier.go:830
github.com/ngaut/unistore/tikv/raftstore.(*applier).applyRaftCmd
/Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/applier.go:758
github.com/ngaut/unistore/tikv/raftstore.(*applier).processRaftCmd
/Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/applier.go:729
github.com/ngaut/unistore/tikv/raftstore.(*applier).handleRaftEntryNormal
/Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/applier.go:647
github.com/ngaut/unistore/tikv/raftstore.(*applier).handleRaftCommittedEntries
/Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/applier.go:600
github.com/ngaut/unistore/tikv/raftstore.(*applier).handleApply
/Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/applier.go:1459
github.com/ngaut/unistore/tikv/raftstore.(*applier).handleTask
/Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/applier.go:1552
github.com/ngaut/unistore/tikv/raftstore.(*raftWorker).runApply
/Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/peer_state.go:263
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1357
This API is used by TiDB to drop the data of a table or index.
If there are blob files, we need to take care of blob data discard info.
lein run test --workload=bank-multitable --time-limit=60 --txn-mode=pessimistic
When unistore exit unnormally, restart panic again
panic: entries[202629:226250) is unavailable from storage
goroutine 1 [running]:
log.(*Logger).Panicf(0xc000248fa0, 0x5195552, 0x2a, 0xc02ab67360, 0x2, 0x2)
/usr/local/go/src/log/log.go:228 +0xc1
github.com/zhangjinpeng1987/raft.(*DefaultLogger).Panicf(0x60d1bc0, 0x5195552, 0x2a, 0xc02ab67360, 0x2, 0x2)
/Users/zhou/gopath/pkg/mod/github.com/zhangjinpeng1987/[email protected]/logger.go:121 +0x60
github.com/zhangjinpeng1987/raft.(*RaftLog).slice(0xc000345500, 0x31785, 0x373ca, 0xffffffffffffffff, 0x1, 0x0, 0x1, 0xc045af6840, 0x0)
/Users/zhou/gopath/pkg/mod/github.com/zhangjinpeng1987/[email protected]/log.go:337 +0x5a0
github.com/zhangjinpeng1987/raft.(*Raft).Step(0xc0000d6a00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/Users/zhou/gopath/pkg/mod/github.com/zhangjinpeng1987/[email protected]/raft.go:957 +0x95
github.com/zhangjinpeng1987/raft.(*RawNode).Campaign(...)
/Users/zhou/gopath/pkg/mod/github.com/zhangjinpeng1987/[email protected]/rawnode.go:142
github.com/ngaut/unistore/tikv/raftstore.NewPeer(0x1, 0xc000444600, 0xc0003c3f50, 0xc00017e680, 0xc02ab6e1e0, 0xc00042b780, 0x2, 0xc000072c00, 0x7940108)
/Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/peer.go:358 +0x785
github.com/ngaut/unistore/tikv/raftstore.createPeerFsm(0x1, 0xc000444600, 0xc02ab6e1e0, 0xc0003c3f50, 0xc00017e680, 0x0, 0x0, 0xc045af3238)
/Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/fsm_peer.go:72 +0x141
github.com/ngaut/unistore/tikv/raftstore.(*raftBatchSystem).loadPeers.func1(0xc02ab70090, 0x0, 0x0)
/Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/fsm_store.go:283 +0x587
github.com/coocood/badger.(*DB).View(0xc0002f0900, 0xc006ead7e0, 0x0, 0x0)
/Users/zhou/gopath/pkg/mod/github.com/coocood/[email protected]/transaction.go:568 +0x80
github.com/ngaut/unistore/tikv/raftstore.(*raftBatchSystem).loadPeers(0xc00018a870, 0x0, 0x0, 0x0, 0x0, 0x0)
/Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/fsm_store.go:240 +0x35b
github.com/ngaut/unistore/tikv/raftstore.(*raftBatchSystem).start(0xc00018a870, 0xc0002ba250, 0xc000444600, 0xc0003c3f50, 0x5353d40, 0xc0000a2500, 0x53977e0, 0xc006e1e000, 0xc0000a2370, 0xc00018a5d0, ...)
/Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/fsm_store.go:413 +0x674
github.com/ngaut/unistore/tikv/raftstore.(*Node).startNode(0xc0000a2410, 0xc0003c3f50, 0x5353d40, 0xc0000a2500, 0xc0000a2370, 0xc00018a5d0, 0x0, 0x0)
/Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/node.go:251 +0x144
github.com/ngaut/unistore/tikv/raftstore.(*Node).Start(0xc0000a2410, 0x5380e60, 0xc000122008, 0xc0003c3f50, 0x5353d40, 0xc0000a2500, 0xc0000a2370, 0xc00018a5d0, 0xc0001a2380, 0x40665a0, ...)
/Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/node.go:93 +0x197
github.com/ngaut/unistore/tikv/raftstore.(*RaftInnerServer).Start(0xc0002ba240, 0x53977e0, 0xc006e1e000, 0xc0001a2400, 0x5383420)
/Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/server.go:156 +0x2a0
main.setupRaftInnerServer(0xc00021e0c0, 0xc0001231c8, 0x53977e0, 0xc006e1e000, 0xc000102a80, 0x53977e0, 0xc006e1e000, 0x0, 0x0, 0xc)
/Users/zhou/gorepo/src/github.com/ngaut/unistore/unistore-server/main.go:278 +0x6b3
main.main()
/Users/zhou/gorepo/src/github.com/ngaut/unistore/unistore-server/main.go:137 +0x540
unistore/tikv/raftstore/worker.go
Line 641 in 01b39cc
To make unistore work with the latest TiDB.
Ths current unistore source code unit-test coverage is low, try to increase unit-test coverage, and add cases using fail-point utils to increase robustness, and to find problems earlier
? github.com/ngaut/unistore/config [no test files]
ok github.com/ngaut/unistore/lockstore 10.757s coverage: 72.4% of statements
? github.com/ngaut/unistore/metrics [no test files]
? github.com/ngaut/unistore/pd [no test files]
ok github.com/ngaut/unistore/rocksdb 2.756s coverage: 81.6% of statements
ok github.com/ngaut/unistore/rowcodec 0.035s coverage: 42.8% of statements
ok github.com/ngaut/unistore/tikv 0.356s coverage: 17.9% of statements
? github.com/ngaut/unistore/tikv/dbreader [no test files]
? github.com/ngaut/unistore/tikv/mvcc [no test files]
ok github.com/ngaut/unistore/tikv/raftstore 5.653s coverage: 13.3% of statements
? github.com/ngaut/unistore/unistore-server [no test files]
? github.com/ngaut/unistore/util [no test files]
ok github.com/ngaut/unistore/util/lockwaiter 0.106s coverage: 98.6% of statements
It assumes that the raftcmd generated by the transaction write has a specific order, but when ran with tikv, the assumption is not true. For example, for prewrite, the write of lock cf may precede default cf. This causes unikv to fail to replicate data from tikv. In addition, this function is based on txnkv, which also makes it impossible to support rawkv.
unistore/tikv/raftstore/fsm_apply.go
Lines 960 to 1052 in 7d1b4e2
panic: interface conversion: interface {} is nil, not uint64
goroutine 130 [running]:
github.com/ngaut/unistore/tikv/raftstore.(*applier).handleTask(0xc03ceda0c0, 0xc0001c41e0, 0x132, 0x12, 0x0, 0x0)
/Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/applier.go:1558 +0x1f0
github.com/ngaut/unistore/tikv/raftstore.(*raftWorker).runApply(0xc0001470e0, 0xc0001cc6d0)
/Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/peer_state.go:263 +0x147
created by github.com/ngaut/unistore/tikv/raftstore.(*raftWorker).run
/Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/peer_state.go:122 +0x64
show stats_histograms where table_name = 'orders';
expect
+---------+------------+----------------+--------------+----------+---------------------+----------------+------------+--------------+-------------+
| Db_name | Table_name | Partition_name | Column_name | Is_index | Update_time | Distinct_count | Null_count | Avg_col_size | Correlation |
+---------+------------+----------------+--------------+----------+---------------------+----------------+------------+--------------+-------------+
| test | orders | | o_carrier_id | 0 | 2020-04-03 14:41:57 | 10 | 90000 | 6.3 | 0.080504 |
| test | orders | | o_w_id | 0 | 2020-04-03 14:41:57 | 10 | 0 | 9 | 1 |
| test | orders | | o_c_id | 0 | 2020-04-03 14:41:57 | 3000 | 0 | 9 | -0.019076 |
| test | orders | | idx_order | 1 | 2020-04-03 14:41:57 | 300000 | 0 | 0 | 0 |
| test | orders | | PRIMARY | 1 | 2020-04-03 14:41:57 | 300000 | 0 | 0 | 0 |
+---------+------------+----------------+--------------+----------+---------------------+----------------+------------+--------------+-------------+
got
+---------+------------+----------------+-------------+----------+---------------------+----------------+------------+--------------+-------------+
| Db_name | Table_name | Partition_name | Column_name | Is_index | Update_time | Distinct_count | Null_count | Avg_col_size | Correlation |
+---------+------------+----------------+-------------+----------+---------------------+----------------+------------+--------------+-------------+
| test | orders | | idx_order | 1 | 2020-04-03 14:44:49 | 300000 | 0 | 0 | 0 |
| test | orders | | PRIMARY | 1 | 2020-04-03 14:44:49 | 300000 | 0 | 0 | 0 |
+---------+------------+----------------+-------------+----------+---------------------+----------------+------------+--------------+-------------+
An improvement: Do not query bloom filter when hash index is enabled.
2018/06/04 15:54:01.280 terror.go:342: [error] [kv:8]invalid transaction
/home/pingcap/projects/src/github.com/pingcap/tidb/kv/txn.go:67:
2018/06/04 15:54:01.281 txn.go:65: [warning] [kv] Retry txn 400581942642737155 original txn 400581941856305164 err [try again later]: tikv restarts txn: retryable: key is locked, try again later
2018/06/04 15:54:01.281 terror.go:342: [error] [kv:8]invalid transaction
/home/pingcap/projects/src/github.com/pingcap/tidb/kv/txn.go:67:
2018/06/04 15:54:01.285 txn.go:65: [warning] [kv] Retry txn 400581942642737159 original txn 400581941856305158 err [try again later]: tikv restarts txn: retryable: key is locked, try again later
Applying logic seems to has some problem processing prepare_merge messages.
applier.go:838: [info] [region 8] 9 execute admin command. term 6, index 11, command cmd_type:PrepareMerge prepare_merge:<min_index:11 target:<id:24 start_key:"t\000\000\000\000\000\000\000\370" end_key:"t\200\000\000\000\000\000\000\377\005\000\000\000\000\000\000\000\370" region_epoch:<conf_ver:5 version:6 > peers:<id:25 store_id:1 > peers:<id:26 store_id:12 > peers:<id:27 store_id:13 > > >
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xfd4113]
goroutine 120 [running]:
github.com/ngaut/unistore/tikv/raftstore.(*applier).execAdminCmd(0xc011095140, 0xc0001c7f40, 0xc021737cc8, 0x0, 0x0, 0x30, 0x1221f00, 0xc000087b01, 0xc021834000)
/home/rwork/go/src/github.com/pingcap/unistore/tikv/raftstore/applier.go:869 +0xb3
github.com/ngaut/unistore/tikv/raftstore.(*applier).execRaftCmd(0xc011095140, 0xc0001c7f40, 0xc000087cc8, 0x70, 0x70, 0x12e2c40, 0x0, 0xe, 0x58)
/home/rwork/go/src/github.com/pingcap/unistore/tikv/raftstore/applier.go:828 +0x9f
github.com/ngaut/unistore/tikv/raftstore.(*applier).applyRaftCmd(0xc011095140, 0xc0001c7f40, 0xb, 0x6, 0xc000087cc8, 0x0, 0x0, 0x0, 0x10)
/home/rwork/go/src/github.com/pingcap/unistore/tikv/raftstore/applier.go:758 +0xf3
github.com/ngaut/unistore/tikv/raftstore.(*applier).processRaftCmd(0xc011095140, 0xc0001c7f40, 0xb, 0x6, 0xc000087cc8, 0x0, 0x7f84897d4460, 0x0)
/home/rwork/go/src/github.com/pingcap/unistore/tikv/raftstore/applier.go:729 +0xae
github.com/ngaut/unistore/tikv/raftstore.(*applier).handleRaftEntryNormal(0xc011095140, 0xc0001c7f40, 0xc00029e230, 0x0, 0x1, 0xc021788240)
/home/rwork/go/src/github.com/pingcap/unistore/tikv/raftstore/applier.go:647 +0x35a
github.com/ngaut/unistore/tikv/raftstore.(*applier).handleRaftCommittedEntries(0xc011095140, 0xc0001c7f40, 0xc00029e230, 0x1, 0x1)
/home/rwork/go/src/github.com/pingcap/unistore/tikv/raftstore/applier.go:600 +0x49c
github.com/ngaut/unistore/tikv/raftstore.(*applier).handleApply(0xc011095140, 0xc0001c7f40, 0xc0217964b0)
/home/rwork/go/src/github.com/pingcap/unistore/tikv/raftstore/applier.go:1459 +0x9a
github.com/ngaut/unistore/tikv/raftstore.(*applier).handleTask(0xc011095140, 0xc0001c7f40, 0x12d, 0x8, 0x10e5340, 0xc0217964b0)
/home/rwork/go/src/github.com/pingcap/unistore/tikv/raftstore/applier.go:1552 +0x71
github.com/ngaut/unistore/tikv/raftstore.(*raftWorker).runApply(0xc000124ea0, 0xc01b76a010)
/home/rwork/go/src/github.com/pingcap/unistore/tikv/raftstore/peer_state.go:263 +0x147
created by github.com/ngaut/unistore/tikv/raftstore.(*raftWorker).run
/home/rwork/go/src/github.com/pingcap/unistore/tikv/raftstore/peer_state.go:122 +0x64
Current solution:
Region merge not supported yet, use merge-schedule-limit to disable region merge
Currently, in raft mode, region can not be split automatically due to size increase.
size split is not working because ApproximateSize is not supported in badger.
Currently, only one pd-server address is supported.
Support multiple pd-server to tolerate pd-serverr failure.
After stop and restart the node process, some node process crashes
panic: failed to delete key
goroutine 147 [running]:
github.com/ngaut/unistore/tikv/raftstore.(*WriteBatch).WriteToKV(0xc000030ae0, 0xc00000e160, 0x44fcc8, 0x39db729ba1720)
/home/rwork/go/src/github.com/pingcap/unistore/tikv/raftstore/engine.go:255 +0x56d
github.com/ngaut/unistore/tikv/raftstore.(*applyContext).writeToDB(0xc000134b40)
/home/rwork/go/src/github.com/pingcap/unistore/tikv/raftstore/applier.go:344 +0x123
github.com/ngaut/unistore/tikv/raftstore.(*applyContext).flush(0xc000134b40)
/home/rwork/go/src/github.com/pingcap/unistore/tikv/raftstore/applier.go:405 +0x63
github.com/ngaut/unistore/tikv/raftstore.(*raftWorker).runApply(0xc000030b40, 0xc000040130)
/home/rwork/go/src/github.com/pingcap/unistore/tikv/raftstore/peer_state.go:265 +0x172
created by github.com/ngaut/unistore/tikv/raftstore.(*raftWorker).run
/home/rwork/go/src/github.com/pingcap/unistore/tikv/raftstore/peer_state.go:122 +0x64
panic: runtime error: slice bounds out of range
goroutine 434149 [running]:
github.com/ngaut/faketikv/tikv.(*MVCCStore).gcDelKeysInBatch(0xc4202640c0, 0xc426256000, 0xa65, 0xc00, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/pingcap/projects/src/github.com/ngaut/faketikv/tikv/mvcc.go:773 +0x238
github.com/ngaut/faketikv/tikv.(*MVCCStore).gcDelAndRollbacks(0xc4202640c0, 0xc437ed3dc0, 0x1b, 0x20, 0xc44ec6f540, 0x1b, 0x20, 0x58d25a2ae540000, 0x0, 0x0)
/home/pingcap/projects/src/github.com/ngaut/faketikv/tikv/mvcc.go:765 +0x24d
github.com/ngaut/faketikv/tikv.(*MVCCStore).GC(0xc4202640c0, 0xc437ed3dc0, 0x1b, 0x20, 0xc44ec6f540, 0x1b, 0x20, 0x58d25a2ae540000, 0x0, 0xdf5aa0)
/home/pingcap/projects/src/github.com/ngaut/faketikv/tikv/mvcc.go:694 +0x122
github.com/ngaut/faketikv/tikv.(*Server).KvGC(0xc4202640c0, 0xf6f6c0, 0xc47766c390, 0xc479aea570, 0xc4202640c0, 0xc47766c360, 0xd51c40)
/home/pingcap/projects/src/github.com/ngaut/faketikv/tikv/server.go:214 +0x24b
github.com/pingcap/kvproto/pkg/tikvpb._Tikv_KvGC_Handler(0xe8f260, 0xc4202640c0, 0xf6f6c0, 0xc47766c390, 0xc42e265810, 0x0, 0x0, 0x0, 0x4c04ec, 0xc433694d81)
/home/pingcap/projects/src/github.com/pingcap/kvproto/pkg/tikvpb/tikvpb.pb.go:652 +0x241
google.golang.org/grpc.(*Server).processUnaryRPC(0xc420232380, 0xf75b40, 0xc4258dc000, 0xc64f6b11d0, 0xc420382750, 0x16f4850, 0x0, 0x0, 0x0)
/home/pingcap/projects/src/google.golang.org/grpc/server.go:1006 +0x481
google.golang.org/grpc.(*Server).handleStream(0xc420232380, 0xf75b40, 0xc4258dc000, 0xc64f6b11d0, 0x0)
/home/pingcap/projects/src/google.golang.org/grpc/server.go:1243 +0x1318
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc425718aa0, 0xc420232380, 0xf75b40, 0xc4258dc000, 0xc64f6b11d0)
/home/pingcap/projects/src/google.golang.org/grpc/server.go:678 +0x9f
created by google.golang.org/grpc.(*Server).serveStreams.func1
/home/pingcap/projects/src/google.golang.org/grpc/server.go:676 +0xa1
The memory allocated by MemTable and table index is not released.
analyze more than 10 million rows table on unistore using analyze table xxx, the tidb rpc batch client keeps reporting
[WARN] [client_batch.go:594] ["wait response is cancelled"] [to=172.16.5.73:29391] [cause="context deadline exceeded"]
and the analyze query seems got stuck
unistore analyze processing stack
goroutine 96986590 [semacquire]:
sync.runtime_SemacquireMutex(0xc0000c8244, 0x0, 0x1)
/usr/local/go/src/runtime/sema.go:71 +0x47
sync.(*Mutex).lockSlow(0xc0000c8240)
/usr/local/go/src/sync/mutex.go:138 +0xfc
sync.(*Mutex).Lock(...)
/usr/local/go/src/sync/mutex.go:81
math/rand.(*lockedSource).Int63(0xc0000c8240, 0x1a00)
/usr/local/go/src/math/rand/rand.go:388 +0xe5
math/rand.(*Rand).Int63(...)
/usr/local/go/src/math/rand/rand.go:85
math/rand.(*Rand).Int63n(0xc0000f4150, 0xb52f5, 0x2)
/usr/local/go/src/math/rand/rand.go:117 +0x6a
math/rand.Int63n(...)
/usr/local/go/src/math/rand/rand.go:327
github.com/pingcap/tidb/statistics.(*SampleCollector).collect(0xc010d04060, 0xc028df5680, 0x6, 0x0, 0xc02ff1e5e8, 0x2, 0x1a18, 0x0, 0x0, 0x0, ...)
/home/tidb/go/pkg/mod/github.com/pingcap/[email protected]/statistics/sample.go:168 +0x25e
github.com/pingcap/tidb/statistics.SampleBuilder.CollectColumnStats(0xc028df5680, 0x16e6fc0, 0xc007b1aa00, 0xa, 0x0, 0x100, 0x3e8, 0x2710, 0x80000000005, 0x13, ...)
/home/tidb/go/pkg/mod/github.com/pingcap/[email protected]/statistics/sample.go:244 +0x3fd
github.com/ngaut/unistore/tikv.(*Server).handleAnalyzeColumnsReq(0xc000209680, 0xc002b27490, 0xc01ebe2580, 0x13, 0x1b, 0xc01ebe25a0, 0x13, 0x1b, 0xc0297e9a90, 0xffffffffffffffff, ...)
/data1/rwork/src/unistore/tikv/analyze.go:202 +0x667
github.com/ngaut/unistore/tikv.(*Server).handleCopAnalyzeRequest(0xc000209680, 0xc002b27490, 0xc01a933900, 0xb)
/data1/rwork/src/unistore/tikv/analyze.go:75 +0x3bf
github.com/ngaut/unistore/tikv.(*Server).Coprocessor(0xc000209680, 0x16e6e00, 0xc00046a1e0, 0xc01a933900, 0x0, 0x0, 0x0)
/data1/rwork/src/unistore/tikv/server.go:508 +0x1ec
github.com/ngaut/unistore/tikv.(*Server).handleBatchRequest(0xc000209680, 0x16e6e00, 0xc00046a1e0, 0xc01722d860, 0x0, 0xc0317bb622, 0x1a)
/data1/rwork/src/unistore/tikv/server_batch.go:271 +0xa0e
github.com/ngaut/unistore/tikv.(*batchRequestHandler).handleRequest(0xc00046a210, 0x1721707, 0xc01722d860)
/data1/rwork/src/unistore/tikv/server_batch.go:71 +0x54
created by github.com/ngaut/unistore/tikv.(*batchRequestHandler).dispatchBatchRequest
/data1/rwork/src/unistore/tikv/server_batch.go:97 +0x56
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x103bc2f]
goroutine 94 [running]:
github.com/ngaut/unistore/tikv.(*regionCtx).updateRegionEpoch(...)
/Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/region.go:80
github.com/ngaut/unistore/tikv.(*RaftRegionManager).OnRegionConfChange(0xc000090ea0, 0xc000312700, 0xc03d2ba870)
/Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/region.go:333 +0x7f
github.com/ngaut/unistore/tikv/raftstore.(*peerFsmDelegate).onReadyChangePeer(0xc0003a1e90, 0xc0001bda90, 0xc000085800, 0xc03d326200)
/Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/fsm_peer.go:708 +0x22a
github.com/ngaut/unistore/tikv/raftstore.(*peerFsmDelegate).onReadyResult(0xc0003a1e90, 0x0, 0xc0002074e0, 0x1, 0x1, 0x2, 0x2, 0x40c1d6, 0xc0001305a0)
/Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/fsm_peer.go:945 +0x40b
github.com/ngaut/unistore/tikv/raftstore.(*peerFsmDelegate).onApplyResult(0xc0003a1e90, 0xc000178ee0)
/Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/fsm_peer.go:371 +0x17a
github.com/ngaut/unistore/tikv/raftstore.(*peerFsmDelegate).handleMsgs(0xc0003a1e90, 0xc0003a1ea0, 0x1, 0x1)
/Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/fsm_peer.go:154 +0x780
github.com/ngaut/unistore/tikv/raftstore.(*raftWorker).run(0xc00007af00, 0xc0000dcd20, 0xc0000da590)
/Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/peer_state.go:153 +0x9b9
created by github.com/ngaut/unistore/tikv/raftstore.(*raftBatchSystem).startSystem
/Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/fsm_store.go:510 +0x160
Related PR in TiKV
tikv/tikv#4085
MvccGetByKey and MvccGetByStartTS are used to diagnose transaction bug.
unistore/tikv/raftstore/peer_storage.go
Line 214 in 01b39cc
lein run test --workload=bank-multitable --read-lock=update --time-limit 120 --concurrency 2n --txn-mode “pessimistic”
There is a error while i am preparing sysbench data.
tidb log:
"INSERT INTO sbtest1(k, c, pad) VALUES(14947141, '25942538382-19136984219-03861106288-00576970577-24361311750-47763414368-95178035149-83010171818-94394022182-87430334962', '24869299864-61603159965-38945076374-93777316936-50603219421'),(17545198, '81442642794-60382140656-21233555113-58218451272-77109691109-52814287258-89440595380-15163701954-67119216642-43657914142', '24695893933-87471404260-82711175980-59472878298-42570363340'),(15086200, '50221825239-02409764394-42446296722-69370192650-86436920893-85303191233-28256048222-04772872232-95794565156-15075269747', '05086997714-95097113848-83724660080-80357869664-75330299471'),(15137543, '65789183760-12848267816-07060840582-63213013982-02522273917-24637380381-28838778300-85018632103-02165466915-05385930764', '21814949874-44174266203-35420832017-32226301537-92609885410'),(14968754, '33228637159-82746513466-46082047040-52794191177-26669469944-04204687379-09046206646-53573703440-96153431980-67158704032', '97913131593-64960156309-97561504476-56948131188-42597317139'),(15075178, '03368274570-55350378630-46248612450-75075447369-07022025604-68767005599-53220779465-25810639785-89014302611-54509261924', '99845394751-78058218736-57962540933-32604856768-33986950361'),(15001481, '66603183228-61466811240-11430085314-35408672269-57781845116-78008740514-23898105495-88328529221-98508124765-84880427463', '09990774584-19231381853-31659921542-80299963918-50651055923'),(15116754, '68467023426-31773815428-35642348859-05304725239-97985872096-94955178075-43798513426-34301445307-70171762665-48131041859', '39938311449-94586015486-98894138469-93369481986-28174115680'),(15052370, '10289294954-56634955769-95468129144-08111915664-17205037012-11944634561-23902737161-06120887916-03085184108-70366891026', '46832298697-56528320326-63946001979-84556253988-48414140497'),(13173103, '07808623488-37494438529-68911101408-65363181328-23295325189-55770754116-93488292778-92554158790-54359946342-25095094013', '17089085009-56376586045-09234257250-51520035242-83530051065'),(15018422, '65874745031-59894551247-84102470725-18673753548-68140362227-83701094413-50646338769-04077965417-50973078597-71928423289', '34747416711-11257577034-01168384928-98471901351-99679325341'),(15135426, '54826917340-81643664845-21025768793-62303450818-36816204091-61712155967-82956993568-77563968558-12219448499-28883230850', '18058955963-89167631334-64235806479-66208235083-08769055107'),(14978944, '44851766122-91800362170-06708311531-55473388114-96891145540-92457994050-58087050558-66019874438-68779540220-37355423707', '53986375378-96477923848-88510077062-42719080048-91461529042'),(15060061, '10988706069-47734912624-48802843488-92196119291-48788266084-82133788557-15423510118-92506461721-11243795397-04604440472', '47331977007-28355576295-20897049439-53874396831-63935693427'),(15028799, '19604005726-07919770045-98811397774-57830225528-70964553689-58186730194-12857850067-73427305289-62395733770-58988071928', '03139346131-68675015580-76536579821-27819894476-36474540175'),(15076427, '39944082574-94205332313-11017896328-56416004299-63606324792-08465974875-50721995044-99446775127-61652722929-97428107215', '72650417728-76600872741-75898884843-32973569499-04141454205'),(14983607, '70808081401-23107603448-32179536288-75379925487-24522202316-43019653721-97879630877-81725689913-92603905729-62839443825', '56217309847-32941981176-93715183847-62509676499-13611858626'),(15064966, '66899214066-76230744367-60784889735-15838961985-93688473907-02603433152-17996095728-50995360787-66123509673-93954391064', '75150680880-37899821798-15231898535-10222643760-25627465724'),(15110017, '95496239493-57619321091-96548297340-90082026065-56401021560-33177452711-33439288304-77992884274-05838973161-15571979998', '52408365398-14311354607-93268359479-06012850725-80187305298'),(15033672, '26704635751-33895996446-78478356895-13631098690-59711214978-81444225220-38439575626-40843743030-19499305676-20421342416', '70286985010-04405373975-93857133445-27725838956-10699676966'),(14952796, '85230707594-52126585305-89659319998-61006944379-19946243091-30407077239-52427799210-97072746003-91330573852(len: 524232)"
/home/pingcap/projects/src/github.com/pingcap/tidb/plan/optimizer.go:77: privilege check fail
/home/pingcap/projects/src/github.com/pingcap/tidb/executor/compiler.go:50:
/home/pingcap/projects/src/github.com/pingcap/tidb/session/session.go:837:
/home/pingcap/projects/src/github.com/pingcap/tidb/session/session.go:757:
/home/pingcap/projects/src/github.com/pingcap/tidb/server/conn.go:867:
2018/07/11 11:56:48.919 server.go:312: [info] [con:3] close connection
2018/07/11 11:56:52.321 session.go:836: [warning] [con:11] compile error:
privilege check fail
select count() from sbtest1
2018/07/11 11:56:52.321 conn.go:488: [warning] [con:11] dispatch error:
id:11, addr:192.168.198.98:52358 status:2, collation:utf8_general_ci, user:root
"select count() from sbtest1"
/home/pingcap/projects/src/github.com/pingcap/tidb/plan/optimizer.go:77: privilege check fail
/home/pingcap/projects/src/github.com/pingcap/tidb/executor/compiler.go:50:
/home/pingcap/projects/src/github.com/pingcap/tidb/session/session.go:837:
/home/pingcap/projects/src/github.com/pingcap/tidb/session/session.go:757:
/home/pingcap/projects/src/github.com/pingcap/tidb/server/conn.go:867:
Inconsistency happens on the table bmsql_new_order, the handle seems to have fewer data compared with the primary key index
admin check table bmsql_new_order;
ERROR 1105 (HY000): handle 128987, index:types.Datum{k:0x1, collation:0x0, decimal:0x0, length:0x0, i:11, b:[]uint8(nil), x:interface {}(nil)} != record:<nil>
select count(*) from bmsql_new_order;
+----------+
| count(*) |
+----------+
| 452721 |
+----------+
1 row in set (0.13 sec)
select count(*) from bmsql_new_order use index(primary);
+----------+
| count(*) |
+----------+
| 474515 |
+----------+
1 row in set (0.14 sec)
goroutine profile: total 4304
3122 @ 0x40337e0 0x4043dd0 0x4043dbb 0x4043b37 0x4c2027a 0x4c20222 0x4c2090b 0x4c23bbf 0x4c2575c 0x4c2974f 0x4c28e24 0x4061741
# 0x4043b36 sync.runtime_SemacquireMutex+0x46 /usr/local/go/src/runtime/sema.go:71
# 0x4c20279 sync.(*RWMutex).RLock+0x479 /usr/local/go/src/sync/rwmutex.go:50
# 0x4c20221 github.com/ngaut/unistore/tikv.(*regionManager).GetRegionFromCtx+0x421 /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/region.go:215
# 0x4c2090a github.com/ngaut/unistore/tikv.(*RaftRegionManager).GetRegionFromCtx+0x3a /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/region.go:308
# 0x4c23bbe github.com/ngaut/unistore/tikv.newRequestCtx+0xee /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/server.go:88
# 0x4c2575b github.com/ngaut/unistore/tikv.(*Server).KvPrewrite+0x7b /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/server.go:248
# 0x4c2974e github.com/ngaut/unistore/tikv.(*Server).handleBatchRequest+0x1de /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/server_batch.go:141
# 0x4c28e23 github.com/ngaut/unistore/tikv.(*batchRequestHandler).handleRequest+0x53 /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/server_batch.go:58
602 @ 0x40337e0 0x4043dd0 0x4043dbb 0x4043b37 0x4c2027a 0x4c20222 0x4c2090b 0x4c23bbf 0x4c25a6c 0x4c29b7f 0x4c28e24 0x4061741
# 0x4043b36 sync.runtime_SemacquireMutex+0x46 /usr/local/go/src/runtime/sema.go:71
# 0x4c20279 sync.(*RWMutex).RLock+0x479 /usr/local/go/src/sync/rwmutex.go:50
# 0x4c20221 github.com/ngaut/unistore/tikv.(*regionManager).GetRegionFromCtx+0x421 /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/region.go:215
# 0x4c2090a github.com/ngaut/unistore/tikv.(*RaftRegionManager).GetRegionFromCtx+0x3a /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/region.go:308
# 0x4c23bbe github.com/ngaut/unistore/tikv.newRequestCtx+0xee /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/server.go:88
# 0x4c25a6b github.com/ngaut/unistore/tikv.(*Server).KvCommit+0x7b /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/server.go:263
# 0x4c29b7e github.com/ngaut/unistore/tikv.(*Server).handleBatchRequest+0x60e /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/server_batch.go:147
# 0x4c28e23 github.com/ngaut/unistore/tikv.(*batchRequestHandler).handleRequest+0x53 /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/server_batch.go:58
219 @ 0x40337e0 0x4043dd0 0x4043dbb 0x4043a22 0x40711f4 0x4bb10ba 0x4c1a161 0x4c19c23 0x4c2590c 0x4c2974f 0x4c28e24 0x4061741
# 0x4043a21 sync.runtime_Semacquire+0x41 /usr/local/go/src/runtime/sema.go:56
# 0x40711f3 sync.(*WaitGroup).Wait+0x63 /usr/local/go/src/sync/waitgroup.go:130
# 0x4bb10b9 github.com/ngaut/unistore/tikv/raftstore.(*raftDBWriter).Write+0x219 /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/db_writer.go:168
# 0x4c1a160 github.com/ngaut/unistore/tikv.(*MVCCStore).prewriteOptimistic+0x4e0 /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/mvcc.go:321
# 0x4c19c22 github.com/ngaut/unistore/tikv.(*MVCCStore).Prewrite+0x1f2 /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/mvcc.go:281
# 0x4c2590b github.com/ngaut/unistore/tikv.(*Server).KvPrewrite+0x22b /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/server.go:256
# 0x4c2974e github.com/ngaut/unistore/tikv.(*Server).handleBatchRequest+0x1de /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/server_batch.go:141
# 0x4c28e23 github.com/ngaut/unistore/tikv.(*batchRequestHandler).handleRequest+0x53 /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/server_batch.go:58
198 @ 0x40337e0 0x4043dd0 0x4043dbb 0x4043a22 0x40711f4 0x4c1fdd5 0x4c2010d 0x4c2090b 0x4c23bbf 0x4c2575c 0x4c2974f 0x4c28e24 0x4061741
# 0x4043a21 sync.runtime_Semacquire+0x41 /usr/local/go/src/runtime/sema.go:56
# 0x40711f3 sync.(*WaitGroup).Wait+0x63 /usr/local/go/src/sync/waitgroup.go:130
# 0x4c1fdd4 github.com/ngaut/unistore/tikv.(*regionCtx).waitParent+0x44 /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/region.go:182
# 0x4c2010c github.com/ngaut/unistore/tikv.(*regionManager).GetRegionFromCtx+0x30c /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/region.go:245
# 0x4c2090a github.com/ngaut/unistore/tikv.(*RaftRegionManager).GetRegionFromCtx+0x3a /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/region.go:308
# 0x4c23bbe github.com/ngaut/unistore/tikv.newRequestCtx+0xee /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/server.go:88
# 0x4c2575b github.com/ngaut/unistore/tikv.(*Server).KvPrewrite+0x7b /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/server.go:248
# 0x4c2974e github.com/ngaut/unistore/tikv.(*Server).handleBatchRequest+0x1de /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/server_batch.go:141
# 0x4c28e23 github.com/ngaut/unistore/tikv.(*batchRequestHandler).handleRequest+0x53 /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/server_batch.go:58
85 @ 0x40337e0 0x4043dd0 0x4043dbb 0x4043a22 0x40711f4 0x4c1fdd5 0x4c2010d 0x4c2090b 0x4c23bbf 0x4c25a6c 0x4c29b7f 0x4c28e24 0x4061741
# 0x4043a21 sync.runtime_Semacquire+0x41 /usr/local/go/src/runtime/sema.go:56
# 0x40711f3 sync.(*WaitGroup).Wait+0x63 /usr/local/go/src/sync/waitgroup.go:130
# 0x4c1fdd4 github.com/ngaut/unistore/tikv.(*regionCtx).waitParent+0x44 /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/region.go:182
# 0x4c2010c github.com/ngaut/unistore/tikv.(*regionManager).GetRegionFromCtx+0x30c /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/region.go:245
# 0x4c2090a github.com/ngaut/unistore/tikv.(*RaftRegionManager).GetRegionFromCtx+0x3a /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/region.go:308
# 0x4c23bbe github.com/ngaut/unistore/tikv.newRequestCtx+0xee /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/server.go:88
# 0x4c25a6b github.com/ngaut/unistore/tikv.(*Server).KvCommit+0x7b /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/server.go:263
# 0x4c29b7e github.com/ngaut/unistore/tikv.(*Server).handleBatchRequest+0x60e /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/server_batch.go:147
# 0x4c28e23 github.com/ngaut/unistore/tikv.(*batchRequestHandler).handleRequest+0x53 /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/server_batch.go:58
21 @ 0x40337e0 0x4043dd0 0x4043dbb 0x4043b37 0x4c2027a 0x4c20222 0x4c2090b 0x4c23bbf 0x4c23f4c 0x4c29905 0x4c28e24 0x4061741
# 0x4043b36 sync.runtime_SemacquireMutex+0x46 /usr/local/go/src/runtime/sema.go:71
# 0x4c20279 sync.(*RWMutex).RLock+0x479 /usr/local/go/src/sync/rwmutex.go:50
# 0x4c20221 github.com/ngaut/unistore/tikv.(*regionManager).GetRegionFromCtx+0x421 /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/region.go:215
# 0x4c2090a github.com/ngaut/unistore/tikv.(*RaftRegionManager).GetRegionFromCtx+0x3a /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/region.go:308
# 0x4c23bbe github.com/ngaut/unistore/tikv.newRequestCtx+0xee /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/server.go:88
# 0x4c23f4b github.com/ngaut/unistore/tikv.(*Server).KvGet+0x7b /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/server.go:114
# 0x4c29904 github.com/ngaut/unistore/tikv.(*Server).handleBatchRequest+0x394 /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/server_batch.go:117
# 0x4c28e23 github.com/ngaut/unistore/tikv.(*batchRequestHandler).handleRequest+0x53 /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/server_batch.go:58
8 @ 0x40337e0 0x40085c8 0x400859e 0x400828b 0x4c0a863 0x4061741
# 0x4c0a862 github.com/ngaut/unistore/tikv/raftstore.(*worker).start.func1+0x102 /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/worker.go:205
6 @ 0x40337e0 0x4043dd0 0x4043dbb 0x4043b37 0x4c2027a 0x4c20222 0x4c2090b 0x4c23bbf 0x4c27dec 0x4c29e74 0x4c28e24 0x4061741
# 0x4043b36 sync.runtime_SemacquireMutex+0x46 /usr/local/go/src/runtime/sema.go:71
# 0x4c20279 sync.(*RWMutex).RLock+0x479 /usr/local/go/src/sync/rwmutex.go:50
# 0x4c20221 github.com/ngaut/unistore/tikv.(*regionManager).GetRegionFromCtx+0x421 /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/region.go:215
# 0x4c2090a github.com/ngaut/unistore/tikv.(*RaftRegionManager).GetRegionFromCtx+0x3a /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/region.go:308
# 0x4c23bbe github.com/ngaut/unistore/tikv.newRequestCtx+0xee /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/server.go:88
# 0x4c27deb github.com/ngaut/unistore/tikv.(*Server).Coprocessor+0x7b /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/server.go:475
# 0x4c29e73 github.com/ngaut/unistore/tikv.(*Server).handleBatchRequest+0x903 /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/server_batch.go:243
# 0x4c28e23 github.com/ngaut/unistore/tikv.(*batchRequestHandler).handleRequest+0x53 /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/server_batch.go:58
3 @ 0x40337e0 0x4043dd0 0x4043dbb 0x4043b37 0x4c2027a 0x4c20222 0x4c2090b 0x4c23bbf 0x4c2613c 0x4c2a165 0x4c28e24 0x4061741
# 0x4043b36 sync.runtime_SemacquireMutex+0x46 /usr/local/go/src/runtime/sema.go:71
# 0x4c20279 sync.(*RWMutex).RLock+0x479 /usr/local/go/src/sync/rwmutex.go:50
# 0x4c20221 github.com/ngaut/unistore/tikv.(*regionManager).GetRegionFromCtx+0x421 /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/region.go:215
# 0x4c2090a github.com/ngaut/unistore/tikv.(*RaftRegionManager).GetRegionFromCtx+0x3a /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/region.go:308
# 0x4c23bbe github.com/ngaut/unistore/tikv.newRequestCtx+0xee /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/server.go:88
# 0x4c2613b github.com/ngaut/unistore/tikv.(*Server).KvBatchGet+0x7b /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/server.go:305
# 0x4c2a164 github.com/ngaut/unistore/tikv.(*Server).handleBatchRequest+0xbf4 /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/server_batch.go:159
# 0x4c28e23 github.com/ngaut/unistore/tikv.(*batchRequestHandler).handleRequest+0x53 /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/server_batch.go:58
2 @ 0x40337e0 0x40085c8 0x400859e 0x400828b 0x4be386f 0x4061741
# 0x4be386e github.com/ngaut/unistore/tikv/raftstore.(*raftWorker).runApply+0x1ee /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/peer_state.go:245
2 @ 0x40337e0 0x40085c8 0x400859e 0x40082cb 0x44655da 0x4061741
# 0x44655d9 github.com/coocood/badger.(*DB).runFlushMemTable+0xe9 /Users/zhou/gopath/pkg/mod/github.com/coocood/[email protected]/db.go:767
2 @ 0x40337e0 0x40085c8 0x400859e 0x40082cb 0x447f75a 0x4061741
# 0x447f759 github.com/coocood/badger.(*writeWorker).runWriteLSM+0x1a9 /Users/zhou/gopath/pkg/mod/github.com/coocood/[email protected]/writer.go:132
2 @ 0x40337e0 0x40085c8 0x400859e 0x40082cb 0x447f859 0x4061741
# 0x447f858 github.com/coocood/badger.(*writeWorker).runMergeLSM+0x88 /Users/zhou/gopath/pkg/mod/github.com/coocood/[email protected]/writer.go:145
2 @ 0x40337e0 0x404326b 0x4466337 0x4061741
# 0x4466336 github.com/coocood/badger.(*DB).updateSize+0x156 /Users/zhou/gopath/pkg/mod/github.com/coocood/[email protected]/db.go:884
2 @ 0x40337e0 0x404326b 0x446e89d 0x4061741
# 0x446e89c github.com/coocood/badger.(*levelsController).runWorker+0x18c /Users/zhou/gopath/pkg/mod/github.com/coocood/[email protected]/levels.go:196
2 @ 0x40337e0 0x404326b 0x447ede6 0x4061741
# 0x447ede5 github.com/coocood/badger.(*writeWorker).runFlusher+0x115 /Users/zhou/gopath/pkg/mod/github.com/coocood/[email protected]/writer.go:66
2 @ 0x40337e0 0x404326b 0x447f177 0x4061741
# 0x447f176 github.com/coocood/badger.(*writeWorker).runWriteVLog+0x166 /Users/zhou/gopath/pkg/mod/github.com/coocood/[email protected]/writer.go:86
2 @ 0x40337e0 0x404326b 0x4482cee 0x4061741
# 0x4482ced github.com/coocood/badger.Open.func4+0x15d /Users/zhou/gopath/pkg/mod/github.com/coocood/[email protected]/db.go:267
2 @ 0x40337e0 0x4043dd0 0x4043dbb 0x4043a22 0x40711f4 0x4c1fdd5 0x4c2010d 0x4c2090b 0x4c23bbf 0x4c27dec 0x4c29e74 0x4c28e24 0x4061741
# 0x4043a21 sync.runtime_Semacquire+0x41 /usr/local/go/src/runtime/sema.go:56
# 0x40711f3 sync.(*WaitGroup).Wait+0x63 /usr/local/go/src/sync/waitgroup.go:130
# 0x4c1fdd4 github.com/ngaut/unistore/tikv.(*regionCtx).waitParent+0x44 /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/region.go:182
# 0x4c2010c github.com/ngaut/unistore/tikv.(*regionManager).GetRegionFromCtx+0x30c /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/region.go:245
# 0x4c2090a github.com/ngaut/unistore/tikv.(*RaftRegionManager).GetRegionFromCtx+0x3a /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/region.go:308
# 0x4c23bbe github.com/ngaut/unistore/tikv.newRequestCtx+0xee /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/server.go:88
# 0x4c27deb github.com/ngaut/unistore/tikv.(*Server).Coprocessor+0x7b /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/server.go:475
# 0x4c29e73 github.com/ngaut/unistore/tikv.(*Server).handleBatchRequest+0x903 /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/server_batch.go:243
# 0x4c28e23 github.com/ngaut/unistore/tikv.(*batchRequestHandler).handleRequest+0x53 /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/server_batch.go:58
1 @ 0x402f3f2 0x4047f46 0x4387212 0x4061741
# 0x4047f45 os/signal.signal_recv+0x95 /usr/local/go/src/runtime/sigqueue.go:144
# 0x4387211 os/signal.loop+0x21 /usr/local/go/src/os/signal/signal_unix.go:23
1 @ 0x40337e0 0x402e64a 0x402dc15 0x40d20e5 0x40d30bb 0x40d309d 0x412e9df 0x41415d8 0x41bfd1a 0x4071857 0x45801d7 0x458018a 0x4580a41 0x459e9ee 0x4061741
# 0x402dc14 internal/poll.runtime_pollWait+0x54 /usr/local/go/src/runtime/netpoll.go:184
# 0x40d20e4 internal/poll.(*pollDesc).wait+0x44 /usr/local/go/src/internal/poll/fd_poll_runtime.go:87
# 0x40d30ba internal/poll.(*pollDesc).waitRead+0x22a /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
# 0x40d309c internal/poll.(*FD).Read+0x20c /usr/local/go/src/internal/poll/fd_unix.go:169
# 0x412e9de net.(*netFD).Read+0x4e /usr/local/go/src/net/fd_unix.go:202
# 0x41415d7 net.(*conn).Read+0x67 /usr/local/go/src/net/net.go:184
# 0x41bfd19 bufio.(*Reader).Read+0x269 /usr/local/go/src/bufio/bufio.go:226
# 0x4071856 io.ReadAtLeast+0x86 /usr/local/go/src/io/io.go:310
# 0x45801d6 io.ReadFull+0x86 /usr/local/go/src/io/io.go:329
# 0x4580189 golang.org/x/net/http2.readFrameHeader+0x39 /Users/zhou/gopath/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237
# 0x4580a40 golang.org/x/net/http2.(*Framer).ReadFrame+0xa0 /Users/zhou/gopath/pkg/mod/golang.org/x/[email protected]/http2/frame.go:492
# 0x459e9ed google.golang.org/grpc/internal/transport.(*http2Client).reader+0x17d /Users/zhou/gopath/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1218
1 @ 0x40337e0 0x402e64a 0x402dc15 0x40d20e5 0x40d30bb 0x40d309d 0x412e9df 0x41415d8 0x42e4998 0x4061741
# 0x402dc14 internal/poll.runtime_pollWait+0x54 /usr/local/go/src/runtime/netpoll.go:184
# 0x40d20e4 internal/poll.(*pollDesc).wait+0x44 /usr/local/go/src/internal/poll/fd_poll_runtime.go:87
# 0x40d30ba internal/poll.(*pollDesc).waitRead+0x22a /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
# 0x40d309c internal/poll.(*FD).Read+0x20c /usr/local/go/src/internal/poll/fd_unix.go:169
# 0x412e9de net.(*netFD).Read+0x4e /usr/local/go/src/net/fd_unix.go:202
# 0x41415d7 net.(*conn).Read+0x67 /usr/local/go/src/net/net.go:184
# 0x42e4997 net/http.(*connReader).backgroundRead+0x57 /usr/local/go/src/net/http/server.go:677
1 @ 0x40337e0 0x402e64a 0x402dc15 0x40d20e5 0x40d30bb 0x40d309d 0x412e9df 0x41415d8 0x42e4de4 0x41bf653 0x41c03ad 0x41c05e4 0x426a66c 0x42df3c2 0x42df3f1 0x42e60df 0x42ea664 0x4061741
# 0x402dc14 internal/poll.runtime_pollWait+0x54 /usr/local/go/src/runtime/netpoll.go:184
# 0x40d20e4 internal/poll.(*pollDesc).wait+0x44 /usr/local/go/src/internal/poll/fd_poll_runtime.go:87
# 0x40d30ba internal/poll.(*pollDesc).waitRead+0x22a /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
# 0x40d309c internal/poll.(*FD).Read+0x20c /usr/local/go/src/internal/poll/fd_unix.go:169
# 0x412e9de net.(*netFD).Read+0x4e /usr/local/go/src/net/fd_unix.go:202
# 0x41415d7 net.(*conn).Read+0x67 /usr/local/go/src/net/net.go:184
# 0x42e4de3 net/http.(*connReader).Read+0xf3 /usr/local/go/src/net/http/server.go:785
# 0x41bf652 bufio.(*Reader).fill+0x102 /usr/local/go/src/bufio/bufio.go:100
# 0x41c03ac bufio.(*Reader).ReadSlice+0x3c /usr/local/go/src/bufio/bufio.go:359
# 0x41c05e3 bufio.(*Reader).ReadLine+0x33 /usr/local/go/src/bufio/bufio.go:388
# 0x426a66b net/textproto.(*Reader).readLineSlice+0x6b /usr/local/go/src/net/textproto/reader.go:57
# 0x42df3c1 net/textproto.(*Reader).ReadLine+0x91 /usr/local/go/src/net/textproto/reader.go:38
# 0x42df3f0 net/http.readRequest+0xc0 /usr/local/go/src/net/http/request.go:1012
# 0x42e60de net/http.(*conn).readRequest+0x15e /usr/local/go/src/net/http/server.go:965
# 0x42ea663 net/http.(*conn).serve+0x6d3 /usr/local/go/src/net/http/server.go:1817
1 @ 0x40337e0 0x402e64a 0x402dc15 0x40d20e5 0x40d4aa8 0x40d4a87 0x412f352 0x414a742 0x4149377 0x42ef2d6 0x42eeff7 0x4c341d0 0x4c34165 0x4061741
# 0x402dc14 internal/poll.runtime_pollWait+0x54 /usr/local/go/src/runtime/netpoll.go:184
# 0x40d20e4 internal/poll.(*pollDesc).wait+0x44 /usr/local/go/src/internal/poll/fd_poll_runtime.go:87
# 0x40d4aa7 internal/poll.(*pollDesc).waitRead+0x1f7 /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
# 0x40d4a86 internal/poll.(*FD).Accept+0x1d6 /usr/local/go/src/internal/poll/fd_unix.go:384
# 0x412f351 net.(*netFD).accept+0x41 /usr/local/go/src/net/fd_unix.go:238
# 0x414a741 net.(*TCPListener).accept+0x31 /usr/local/go/src/net/tcpsock_posix.go:139
# 0x4149376 net.(*TCPListener).Accept+0x46 /usr/local/go/src/net/tcpsock.go:261
# 0x42ef2d5 net/http.(*Server).Serve+0x285 /usr/local/go/src/net/http/server.go:2896
# 0x42eeff6 net/http.(*Server).ListenAndServe+0xb6 /usr/local/go/src/net/http/server.go:2825
# 0x4c341cf net/http.ListenAndServe+0x10f /usr/local/go/src/net/http/server.go:3080
# 0x4c34164 main.main.func1+0xa4 /Users/zhou/gorepo/src/github.com/ngaut/unistore/unistore-server/main.go:52
1 @ 0x40337e0 0x404326b 0x458e7fa 0x458f003 0x45af97b 0x4061741
# 0x458e7f9 google.golang.org/grpc/internal/transport.(*controlBuffer).get+0x119 /Users/zhou/gopath/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:317
# 0x458f002 google.golang.org/grpc/internal/transport.(*loopyWriter).run+0x1e2 /Users/zhou/gopath/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:435
# 0x45af97a google.golang.org/grpc/internal/transport.newHTTP2Client.func3+0x7a /Users/zhou/gopath/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:328
1 @ 0x40337e0 0x404326b 0x45aca37 0x45ac7fa 0x45ad695 0x4071857 0x45ad5d7 0x45ad597 0x45dc693 0x45dd3bd 0x45dda6b 0x45eceb0 0x45f4556 0x45eb4b0 0x45ec1b3 0x460ce82 0x4655aea 0x4061741
# 0x45aca36 google.golang.org/grpc/internal/transport.(*recvBufferReader).read+0x1b6 /Users/zhou/gopath/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:142
# 0x45ac7f9 google.golang.org/grpc/internal/transport.(*recvBufferReader).Read+0x59 /Users/zhou/gopath/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:131
# 0x45ad694 google.golang.org/grpc/internal/transport.(*transportReader).Read+0x54 /Users/zhou/gopath/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:414
# 0x4071856 io.ReadAtLeast+0x86 /usr/local/go/src/io/io.go:310
# 0x45ad5d6 io.ReadFull+0xc6 /usr/local/go/src/io/io.go:329
# 0x45ad596 google.golang.org/grpc/internal/transport.(*Stream).Read+0x86 /Users/zhou/gopath/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:398
# 0x45dc692 google.golang.org/grpc.(*parser).recvMsg+0x62 /Users/zhou/gopath/pkg/mod/google.golang.org/[email protected]/rpc_util.go:476
# 0x45dd3bc google.golang.org/grpc.recvAndDecompress+0x4c /Users/zhou/gopath/pkg/mod/google.golang.org/[email protected]/rpc_util.go:607
# 0x45dda6a google.golang.org/grpc.recv+0x9a /Users/zhou/gopath/pkg/mod/google.golang.org/[email protected]/rpc_util.go:650
# 0x45eceaf google.golang.org/grpc.(*csAttempt).recvMsg+0xef /Users/zhou/gopath/pkg/mod/google.golang.org/[email protected]/stream.go:864
# 0x45f4555 google.golang.org/grpc.(*clientStream).RecvMsg.func1+0x45 /Users/zhou/gopath/pkg/mod/google.golang.org/[email protected]/stream.go:715
# 0x45eb4af google.golang.org/grpc.(*clientStream).withRetry+0x35f /Users/zhou/gopath/pkg/mod/google.golang.org/[email protected]/stream.go:567
# 0x45ec1b2 google.golang.org/grpc.(*clientStream).RecvMsg+0x102 /Users/zhou/gopath/pkg/mod/google.golang.org/[email protected]/stream.go:714
# 0x460ce81 github.com/pingcap/kvproto/pkg/pdpb.(*pDRegionHeartbeatClient).Recv+0x61 /Users/zhou/gopath/pkg/mod/github.com/pingcap/[email protected]/pkg/pdpb/pdpb.pb.go:4017
# 0x4655ae9 github.com/ngaut/unistore/pd.(*client).receiveRegionHeartbeat+0x79 /Users/zhou/gorepo/src/github.com/ngaut/unistore/pd/client.go:196
1 @ 0x40337e0 0x404326b 0x45ceebf 0x4061741
# 0x45ceebe google.golang.org/grpc.(*ccBalancerWrapper).watcher+0x10e /Users/zhou/gopath/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:122
1 @ 0x40337e0 0x404326b 0x45f4037 0x4061741
# 0x45f4036 google.golang.org/grpc.newClientStream.func5+0xd6 /Users/zhou/gopath/pkg/mod/google.golang.org/[email protected]/stream.go:315
1 @ 0x40337e0 0x404326b 0x465594d 0x4061741
# 0x465594c github.com/ngaut/unistore/pd.(*client).heartbeatStreamLoop+0x36c /Users/zhou/gorepo/src/github.com/ngaut/unistore/pd/client.go:180
1 @ 0x40337e0 0x404326b 0x4655cfa 0x4061741
# 0x4655cf9 github.com/ngaut/unistore/pd.(*client).reportRegionHeartbeat+0x119 /Users/zhou/gorepo/src/github.com/ngaut/unistore/pd/client.go:211
1 @ 0x40337e0 0x404326b 0x4be2a53 0x4061741
# 0x4be2a52 github.com/ngaut/unistore/tikv/raftstore.(*raftWorker).run+0x6d2 /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/peer_state.go:126
1 @ 0x40337e0 0x404326b 0x4be3d44 0x4061741
# 0x4be3d43 github.com/ngaut/unistore/tikv/raftstore.(*balancer).run+0x173 /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/peer_state.go:308
1 @ 0x40337e0 0x404326b 0x4bf0030 0x4061741
# 0x4bf002f github.com/ngaut/unistore/tikv/raftstore.(*lockStoreDumper).run+0xef /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/server.go:157
1 @ 0x40337e0 0x404326b 0x4c005cf 0x4061741
# 0x4c005ce github.com/ngaut/unistore/tikv/raftstore.(*tickDriver).run+0x10e /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/ticker.go:102
1 @ 0x40337e0 0x4043dd0 0x4043dbb 0x4043a22 0x40711f4 0x4bb10ba 0x4c1b7c2 0x4c25c21 0x4c29b7f 0x4c28e24 0x4061741
# 0x4043a21 sync.runtime_Semacquire+0x41 /usr/local/go/src/runtime/sema.go:56
# 0x40711f3 sync.(*WaitGroup).Wait+0x63 /usr/local/go/src/sync/waitgroup.go:130
# 0x4bb10b9 github.com/ngaut/unistore/tikv/raftstore.(*raftDBWriter).Write+0x219 /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/db_writer.go:168
# 0x4c1b7c1 github.com/ngaut/unistore/tikv.(*MVCCStore).Commit+0x441 /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/mvcc.go:477
# 0x4c25c20 github.com/ngaut/unistore/tikv.(*Server).KvCommit+0x230 /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/server.go:272
# 0x4c29b7e github.com/ngaut/unistore/tikv.(*Server).handleBatchRequest+0x60e /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/server_batch.go:147
# 0x4c28e23 github.com/ngaut/unistore/tikv.(*batchRequestHandler).handleRequest+0x53 /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/server_batch.go:58
1 @ 0x40337e0 0x4043dd0 0x4043dbb 0x4043a22 0x40711f4 0x4c1fdd5 0x4c2068b 0x4bc6c0d 0x4bc7945 0x4bc288a 0x4bc0d57 0x4be2d39 0x4061741
# 0x4043a21 sync.runtime_Semacquire+0x41 /usr/local/go/src/runtime/sema.go:56
# 0x40711f3 sync.(*WaitGroup).Wait+0x63 /usr/local/go/src/sync/waitgroup.go:130
# 0x4c1fdd4 github.com/ngaut/unistore/tikv.(*regionCtx).waitParent+0x44 /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/region.go:182
# 0x4c2068a github.com/ngaut/unistore/tikv.(*RaftRegionManager).OnSplitRegion+0x7a /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/region.go:292
# 0x4bc6c0c github.com/ngaut/unistore/tikv/raftstore.(*peerFsmDelegate).onReadySplitRegion+0xc5c /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/fsm_peer.go:877
# 0x4bc7944 github.com/ngaut/unistore/tikv/raftstore.(*peerFsmDelegate).onReadyResult+0x314 /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/fsm_peer.go:952
# 0x4bc2889 github.com/ngaut/unistore/tikv/raftstore.(*peerFsmDelegate).onApplyResult+0x179 /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/fsm_peer.go:370
# 0x4bc0d56 github.com/ngaut/unistore/tikv/raftstore.(*peerFsmDelegate).handleMsgs+0x786 /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/fsm_peer.go:153
# 0x4be2d38 github.com/ngaut/unistore/tikv/raftstore.(*raftWorker).run+0x9b8 /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/peer_state.go:153
1 @ 0x40337e0 0x4043dd0 0x4043dbb 0x4043b37 0x4bd0504 0x4bd02b5 0x4bd054b 0x4bcb8db 0x4bcbbb2 0x4be3ae6 0x4061741
# 0x4043b36 sync.runtime_SemacquireMutex+0x46 /usr/local/go/src/runtime/sema.go:71
# 0x4bd0503 sync.(*RWMutex).RLock+0x2b3 /usr/local/go/src/sync/rwmutex.go:50
# 0x4bd02b4 github.com/ngaut/unistore/tikv/raftstore.(*storeFsmDelegate).storeHeartbeatPD+0x64 /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/fsm_store.go:772
# 0x4bd054a github.com/ngaut/unistore/tikv/raftstore.(*storeFsmDelegate).onPDStoreHearbeatTick+0x2a /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/fsm_store.go:794
# 0x4bcb8da github.com/ngaut/unistore/tikv/raftstore.(*storeFsmDelegate).onTick+0x4a /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/fsm_store.go:208
# 0x4bcbbb1 github.com/ngaut/unistore/tikv/raftstore.(*storeFsmDelegate).handleMessages+0x2a1 /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/fsm_store.go:231
# 0x4be3ae5 github.com/ngaut/unistore/tikv/raftstore.(*storeWorker).run+0x95 /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/raftstore/peer_state.go:286
1 @ 0x40337e0 0x4050937 0x405090d 0x4c23aa9 0x4c32aa6 0x403340e 0x4061741
# 0x405090c time.Sleep+0x12c /usr/local/go/src/runtime/time.go:105
# 0x4c23aa8 github.com/ngaut/unistore/tikv.(*Server).Stop+0x38 /Users/zhou/gorepo/src/github.com/ngaut/unistore/tikv/server.go:61
# 0x4c32aa5 main.main+0x665 /Users/zhou/gorepo/src/github.com/ngaut/unistore/unistore-server/main.go:99
# 0x403340d runtime.main+0x21d /usr/local/go/src/runtime/proc.go:203
1 @ 0x437c675 0x437c490 0x43790ba 0x4385a2a 0x4386441 0x42eba14 0x42ed8ed 0x42eef14 0x42ea805 0x4061741
# 0x437c674 runtime/pprof.writeRuntimeProfile+0x94 /usr/local/go/src/runtime/pprof/pprof.go:708
# 0x437c48f runtime/pprof.writeGoroutine+0x9f /usr/local/go/src/runtime/pprof/pprof.go:670
# 0x43790b9 runtime/pprof.(*Profile).WriteTo+0x3d9 /usr/local/go/src/runtime/pprof/pprof.go:329
# 0x4385a29 net/http/pprof.handler.ServeHTTP+0x339 /usr/local/go/src/net/http/pprof/pprof.go:245
# 0x4386440 net/http/pprof.Index+0x6f0 /usr/local/go/src/net/http/pprof/pprof.go:268
# 0x42eba13 net/http.HandlerFunc.ServeHTTP+0x43 /usr/local/go/src/net/http/server.go:2007
# 0x42ed8ec net/http.(*ServeMux).ServeHTTP+0x1bc /usr/local/go/src/net/http/server.go:2387
# 0x42eef13 net/http.serverHandler.ServeHTTP+0xa3 /usr/local/go/src/net/http/server.go:2802
# 0x42ea804 net/http.(*conn).serve+0x874 /usr/local/go/src/net/http/server.go:1890
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.