Go Open Source, Distributed, Simple and efficient Search Engine; Warning: This is V1 and beta version, because of big memory consume, and the V2 will be rewrite all code.
License: Apache License 2.0
看特征介绍里面说支持分布式。
riot哪些地方支持分布式，如何分布式的， 怎么使用分布式特性？
谢谢！
Support word, excel, ppt, pdf?
是否支持 word, excel, ppt, pdf 等文档的全文搜索，如没有，有没有方案支持
使用的是微博搜索那个例子，将数据改为从自己数据库中搜索，但是写入的速度很慢，每分钟只能写入3000-5000条，已经使用了协程，不知道是不是哪里写错了。代码如下
Using the Weibo search example, the data is changed to search from my own database, but the write speed is very slow, only 3000-5000 can be written per minute. Correspondence has been used,I don't know where it was wrong.Code show as below
for i := 0; i < 100; i++ {
go indexXwz(xwzs)
}
func indexXwz(xwzs <-chan Xwz) {
for xwz := range xwzs {
searcher.IndexDoc(xwz.Id, types.DocIndexData{
Content: xwz.Name,
Fields: XwzScoringFields{
Timestamp: xwz.LatestDate,
CountNum: xwz.CountNum,
},
}, true)
}
searcher.Flush()
}
Id is uint64 or string?
type DocData struct {
...
Attri interface{}
...
}
type parentInfo struct{
Name string
Id int64
}
type resIndexNode struct {
Id int64
Name string
Attr []parentInfo
}
...
searcher.Init(types.EngineOpts{
Using: 3,
UseStore: true,
StoreFolder:dataPath,
})
...
func addIndex(rin*resIndexNode) {
searcher.Index(uint64(rin.Id), types.DocData{Content: rin.Name,Attri:rin.Attr},true)
searcher.Flush()
}
reload index:
func reloadIndex(dataPath string) {
searcher.Init(types.EngineOpts{
Using: 3,
UseStore: true,
StoreFolder:dataPath,
})
searcher.FlushIndex()
log.Println("recover index number: ", searcher.NumDocsIndexed())
}
searcher.NumDocsIndexed() is 0
right now the library logs wherever it wants, which is rather annoying because:
riot whenever a test failsproposal:
doc_index_data.go:20: syntax error: unexpected = in type declaration github.com/go-ego/g
go version go1.8.3 darwin/amd64
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/usr/include/c++/4.2.1
Apple LLVM version 9.1.0 (clang-902.0.39.2)
Target: x86_64-apple-darwin18.0.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
Hi, I am a newbie and trying out riot.
When I run
........github.com/go-ego/riot/examples/codelab]$ go run search_server.go
I found out many Chinese phrase (text/keywords) didn't return any results. For example, I can get results from the keywords 中 and **. But when I tried 中秋, 国庆, 企业, 企业家 etc., there is nothing returned. Then I tried a person name 李小龙, and there is no result either. I also saw the target index document had the context ...... @薛蛮子: ......, then I tried the keyword 薛蛮子, still no result. I don't really understand because the words 薛蛮子 is separated between two punctuation and should be easily searchable from what I feel.
Since I am a newbie, I am wondering what causes these keywords can not return any results. The above keywords are just some examples. I tried more than that. Only small set of Chinese phrase (is that called text segmentation?) can be supported. The most keywords that I tried can not return anything. Is this due to a bad SegmenterDict dictionary.txt or project code?
Btw I haven't read the doc and code in detail, so sorry if I am totally wrong. Thank you very much for the help.
My Environment:
$ dep status
PROJECT CONSTRAINT VERSION REVISION LATEST PKGS USED
github.com/AndreasBriese/bbloom * branch master 28f7e88 28f7e88 1
github.com/boltdb/bolt * v1.3.1 2f1ce7a 2f1ce7a 1
github.com/dgraph-io/badger * v1.0.1 89689ef 89689ef 6
github.com/dgryski/go-farm * branch master e2d0fe2 e2d0fe2 1
github.com/go-ego/cedar * branch master 5ebe674 5ebe674 1
github.com/go-ego/gpy * branch master 66bb7e1 66bb7e1 1
github.com/go-ego/gse * branch master b16cca4 b16cca4 1
github.com/go-ego/murmur * branch master 84b529b 84b529b 1
github.com/go-ego/riot branch master branch master 3431598 3431598 5
github.com/golang/protobuf * branch master 1643683 1643683 1
github.com/golang/snappy * branch master 553a641 553a641 1
github.com/pkg/errors * v0.8.0 645ef00 645ef00 1
github.com/shirou/gopsutil * v2.17.10 48fc561 48fc561 2
github.com/syndtr/goleveldb * branch master 549b6d6 549b6d6 12
golang.org/x/net * branch master a337091 a337091 3
golang.org/x/sys * branch master 75813c6 4b45465 2
Hi,
I have a use case where each object can and will have completely different fields(and data types). So I cannot create schema that the search server can use to index this data in advance.
With elasticsearch I would have to create basically new index for every object which would be a nightmare, or translate each field into flat key/value structure and have a global index of these without a lot of features that would be otherwise available(facets, range search..).
Can riot handle such use case?
This is the output
2017/10/23 15:41:30 Check virtualMemory...
Total: 8551211008, Free:0, UsedPercent:44.000000%
2017/10/23 15:41:30 {[google testing] [] false 0}
I dont think this is the right output.
I want get the value there i have set in the types.DocumentIndexData, bug the engine can only return the docid.
Just wondering why to keep all third lib source code in vendor directory? That case some dependence issue in my workspace, I think keep the vendor.json is enough ? Source code make the repo larger and keep extra source in project vendor.
What I use is glide to config my project, I think it work well and recommend to you ~
#19
can i use
RemoveDoc(id, forceUpdate)
and then
Index(id, types.DocData{Content: title,},forceUpdate)
is that thread safe?
Line 95 in f686b77
Shouldn't either defer the iterator release or explicitly release the iterator here before this early return?
panic: http: multiple registrations for /debug/requests
goroutine 1 [running]:
net/http.(*ServeMux).Handle(0x1681840, 0xf9da88, 0xf, 0x16275a0, 0xfdc2e0)
/usr/local/go/src/net/http/server.go:2270 +0x627
net/http.(*ServeMux).HandleFunc(0x1681840, 0xf9da88, 0xf, 0xfdc2e0)
/usr/local/go/src/net/http/server.go:2302 +0x55
net/http.HandleFunc(0xf9da88, 0xf, 0xfdc2e0)
/usr/local/go/src/net/http/server.go:2314 +0x4b
github.com/go-ego/riot/vendor/golang.org/x/net/trace.init.0()
/home/ivk/go/src/github.com/go-ego/riot/vendor/golang.org/x/net/trace/trace.go:115 +0x42
github.com/go-ego/riot/vendor/golang.org/x/net/trace.init()
:1 +0x1cd
github.com/go-ego/riot/vendor/github.com/dgraph-io/badger.init()
:1 +0x78
github.com/go-ego/riot/storage.init()
This problem resolving very simple -- removing "net" dependency from:
Gopkg.toml.
#[[constraint]]
Any numeric search + multi facet search features?
I would like to have multi faceted search capabilities.
Also, just curious, has anyone done any benchmark of the search performance, disk and memory requirement for riot vs other open source search engine out there?
Thanks in advance
should i user a timer to timely refresh the cache into index?
i see there is a addCache and removeCache which default size is 300000, does it mean only when data size in cache large than 300000, will it add data into index?
When support geo search?
For some reason I have to use more than one engine at the same process, but while init each engine have to load the dictionary for segment, that cost a lot of time and memory, if I share the segmenter for different engine, will that case the concurrent problem?
If it can be done, may I make a PR for this engine option?
...
Multi programming language client and more api.
Hi, in the scoringFields example, you just mentioned that scoringFields can be persistent,but how? We are now using riot as our first level chioce, so please help us
hi,
after set UseStorage: true, I can create index from the StorageFolder, but i still need the original content like stringData in engine.IndexDoc(DocId, types.DocIndexData{Content: stringData}). I have many files and read them from disk is time-comsuming, is there any way to check whether the specific DocId exists and restore the index without stringData(only through StorageFolder)?
thanks!
以前用过Lucen，比如一文章，有标题，内容，创建时间，在建索引时我可以设置标题的权重0.7，内容的权重0.3，创建时间保存，但不参加索引。
不知道用你这个可不可以做到这样的效果。
The persistent storage data restart project disappears
when i run the simplest_example, it shows the error below, which means cant load the dictionary.
"2017/10/30 13:48:28 载入gse词典 github.com/go-ego/riot/data/dict/dictionary.txt
2017/10/30 13:48:28 无法载入字典文件 "github.com/go-ego/riot/data/dict/dictionary.txt"
Hello! I would like to translate the project into Spanish! is it already translated?
如题,谢谢!
(英文水平太差,不好意思)
# github.com/go-ego/riot/net/grpc
src/github.com/go-ego/riot/net/grpc/etcd.go:66:2: undefined: doc.RegisterGreeterServer
src/github.com/go-ego/riot/net/grpc/grpc.go:147:2: undefined: doc.RegisterGreeterServer
src/github.com/go-ego/riot/net/grpc/grpcc.go:52:7: undefined: doc.NewGreeterClient
src/github.com/go-ego/riot/net/grpc/grpcc.go:78:7: undefined: doc.NewGreeterClient
src/github.com/go-ego/riot/net/grpc/grpcc.go:123:7: undefined: doc.NewGreeterClient
# github.com/go-ego/riot/net/heartb
src/github.com/go-ego/riot/net/heartb/heart.go:55:7: undefined: doc.NewGreeterClient
src/github.com/go-ego/riot/net/heartb/heart.go:75:14: undefined: doc.GreeterClient
the updated protobuf generated file does not have the service stubs generated.
see https://smallrepo.com/builds/20180801-180048-dfe5992b for build log
there are a few calls to Fatal throughout the library - while most of them are in low-impact places, this shouldn't be done regardless, functions should return an error for the caller to handle instead
@vcaesar like my other issue i'd be willing to open a PR to implement this as well
How to parse the return value？I just want to get the string I need.
Yacy
也许riot可以成为一个Yacy搜索引擎的Go语言实现？
t.Run("index "+tt.args.object.Hash, func(t *testing.T) {
t.Parallel()
// request index
if err = e.Index(Document{tt.args.object, "", true}); err != nil {
t.Errorf("wanted Index error = false, got %v", err)
}
// we'll be referring to this hash a few times
var objHash = tt.args.object.Hash
// make sure object can be found
if !e.IsIndexed(objHash) {
t.Errorf("wanted IsIndexed = true, got false")
}
// attempt search
if res, err := e.Search(Query{
Text: tt.args.content,
Hashes: []string{objHash},
}); err != nil && len(res) > 0 {
if res[0].Hash != objHash {
t.Errorf("wanted Search to find '%s', but failed", objHash)
}
} else {
t.Errorf("wanted Search to find '%s', but failed", objHash)
}the full code is available at https://github.com/RTradeLtd/Lens/blob/engine/riot%23parallel-tests/engine/engine_parallel_test.go
the race condition appears to be in https://github.com/go-ego/gse/blob/master/hmm/hmm_seg.go#L18, where the library attempts to write to a global map.
# .... snip
fatal error: concurrent map writes
fatal error: concurrent map writes
goroutine 438 [running]:
runtime.throw(0x47ddbb0, 0x15)
/usr/local/Cellar/go/1.11.3/libexec/src/runtime/panic.go:608 +0x72 fp=0xc012007070 sp=0xc012007040 pc=0x402e082
runtime.mapassign(0x46d6480, 0xc0000bfe60, 0x4885433, 0xc00e963cf0)
/usr/local/Cellar/go/1.11.3/libexec/src/runtime/map.go:651 +0x545 fp=0xc0120070f8 sp=0xc012007070 pc=0x40103b5
github.com/RTradeLtd/Lens/vendor/github.com/go-ego/gse/hmm.loadDefEmit()
/Users/robertlin/go/src/github.com/RTradeLtd/Lens/vendor/github.com/go-ego/gse/hmm/prob_emit.go:9 +0xcc fp=0xc012007150 sp=0xc0120070f8 pc=0x4448efc
github.com/RTradeLtd/Lens/vendor/github.com/go-ego/gse/hmm.LoadModel(0x0, 0x0, 0x0)
/Users/robertlin/go/src/github.com/RTradeLtd/Lens/vendor/github.com/go-ego/gse/hmm/hmm_seg.go:28 +0x1b4 fp=0xc012007188 sp=0xc012007150 pc=0x4448134
github.com/RTradeLtd/Lens/vendor/github.com/go-ego/gse.(*Segmenter).LoadModel(0xc000354398, 0x0, 0x0, 0x0)
/Users/robertlin/go/src/github.com/RTradeLtd/Lens/vendor/github.com/go-ego/gse/segmenter.go:107 +0x3f fp=0xc0120071b0 sp=0xc012007188 pc=0x444f81f
github.com/RTradeLtd/Lens/vendor/github.com/go-ego/gse.(*Segmenter).cutDAG(0xc000354398, 0x47d6387, 0xf, 0x0, 0x0, 0x0, 0xc000da0630, 0x3, 0xc000053970)
/Users/robertlin/go/src/github.com/RTradeLtd/Lens/vendor/github.com/go-ego/gse/dag.go:129 +0x6a fp=0xc012007348 sp=0xc0120071b0 pc=0x444bd1a
github.com/RTradeLtd/Lens/vendor/github.com/go-ego/gse.(*Segmenter).Cut(0xc000354398, 0x47d6387, 0xf, 0xc012007490, 0x1, 0x1, 0x240, 0xc0012161e0, 0x1800)
/Users/robertlin/go/src/github.com/RTradeLtd/Lens/vendor/github.com/go-ego/gse/segmenter.go:74 +0xe7 fp=0xc012007408 sp=0xc012007348 pc=0x444f207
github.com/RTradeLtd/Lens/vendor/github.com/go-ego/riot.(*Engine).Segment(0xc000354240, 0x47d6387, 0xf, 0xf, 0xc0003846a8, 0xc000384680)
/Users/robertlin/go/src/github.com/RTradeLtd/Lens/vendor/github.com/go-ego/riot/engine.go:487 +0x2eb fp=0xc0120074c0 sp=0xc012007408 pc=0x445c60b
github.com/RTradeLtd/Lens/vendor/github.com/go-ego/riot.(*Engine).Tokens(0xc000354240, 0x47d6387, 0xf, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/Users/robertlin/go/src/github.com/RTradeLtd/Lens/vendor/github.com/go-ego/riot/engine.go:510 +0x2fc fp=0xc012007550 sp=0xc0120074c0 pc=0x445c92c
github.com/RTradeLtd/Lens/vendor/github.com/go-ego/riot.(*Engine).Search(0xc000354240, 0x47d6387, 0xf, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/Users/robertlin/go/src/github.com/RTradeLtd/Lens/vendor/github.com/go-ego/riot/engine.go:767 +0x9c fp=0xc012007888 sp=0xc012007550 pc=0x445e56c
github.com/RTradeLtd/Lens/engine.(*Engine).Search(0xc01179fb80, 0x47d6387, 0xf, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/Users/robertlin/go/src/github.com/RTradeLtd/Lens/engine/engine.go:283 +0x46b fp=0xc012007de0 sp=0xc012007888 pc=0x463349b
github.com/RTradeLtd/Lens/engine.TestEngine_parallel.func1(0xc000148700)
/Users/robertlin/go/src/github.com/RTradeLtd/Lens/engine/engine_parallel_test.go:76 +0x252 fp=0xc012007fa8 sp=0xc012007de0 pc=0x46362c2
testing.tRunner(0xc000148700, 0xc01128c750)
/usr/local/Cellar/go/1.11.3/libexec/src/testing/testing.go:827 +0xbf fp=0xc012007fd0 sp=0xc012007fa8 pc=0x40f17ff
runtime.goexit()
/usr/local/Cellar/go/1.11.3/libexec/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc012007fd8 sp=0xc012007fd0 pc=0x405ded1
created by testing.(*T).Run
/usr/local/Cellar/go/1.11.3/libexec/src/testing/testing.go:878 +0x353
# ......... snip
For example, what API does business data update through?
When I add the document with same id but different content to the index, the former still can be searched.
Maybe it should be replaced by the latter.
Provide stable Riot API.
Riot is not in the first GOPATH inside.
Hi,, i'm graphic designer and an open source enthusiast. I would like to contribute to your awesome project by proposing logo design for it if you allow me. I'll be doing it as a gift for free to your awesome project.
Best regard. Mirza zulfan.
As the title suggests, how to use go-ego riot in servers w/ mysql as datastore?
比如所有数据a,b；第一批加载了a，第二批还可能会加载到a，也可能加载到b
You MUST delete the content above including this line before posting, otherwise your issue will be invalid.
...
This is a limited approach compared to Lucene/Solr/Elasticsearch LSM which handle high-volume inserts to its indexes with a log-structured merge-tree (LSM) and where the index size is only limited by the available hard disk space.
look https://github.com/huichen/wukong, any difference?
go build -race and ./store in examples/store/...While using go build -race and running with storage:
2018/01/24 16:24:53 Load the gse dictionary ../../data/dict/dictionary.txt
2018/01/24 16:25:15 gse dictionary loaded finished.
==================
WARNING: DATA RACE
Read at 0x000004b4c8c0 by main goroutine:
github.com/go-ego/riot.(*Engine).FlushIndex()
/go/src/github.com/go-ego/riot/engine.go:656 +0x78
main.initEngine()
/go/src/github.com/go-ego/riot/examples/store/main.go:60 +0x378
github.com/go-ego/riot/vendor/github.com/go-ego/gse.(*Segmenter).LoadDict()
/go/src/github.com/go-ego/riot/vendor/github.com/go-ego/gse/segmenter.go:272 +0x35d
github.com/go-ego/riot.(*Engine).Init()
/go/src/github.com/go-ego/riot/engine.go:237 +0x9d2
main.initEngine()
/go/src/github.com/go-ego/riot/examples/store/main.go:34 +0x151
main.main()
/go/src/github.com/go-ego/riot/examples/store/main.go:89 +0x47
Previous write at 0x000004b4c8c0 by goroutine 162:
sync/atomic.AddInt64()
/usr/local/go/src/runtime/race_amd64.s:276 +0xb
github.com/go-ego/riot.(*Engine).indexerAddDocWorker()
/go/src/github.com/go-ego/riot/indexer_worker.go:52 +0x1a2
Goroutine 162 (running) created at:
github.com/go-ego/riot.(*Engine).Init()
/go/src/github.com/go-ego/riot/engine.go:275 +0x831
github.com/go-ego/riot/vendor/github.com/go-ego/gse.(*Segmenter).Read()
/go/src/github.com/go-ego/riot/vendor/github.com/go-ego/gse/segmenter.go:93 +0x64e
github.com/go-ego/riot/vendor/github.com/go-ego/gse.(*Segmenter).LoadDict()
/go/src/github.com/go-ego/riot/vendor/github.com/go-ego/gse/segmenter.go:272 +0x35d
github.com/go-ego/riot.(*Engine).Init()
/go/src/github.com/go-ego/riot/engine.go:237 +0x9d2
main.initEngine()
/go/src/github.com/go-ego/riot/examples/store/main.go:34 +0x151
main.main()
/go/src/github.com/go-ego/riot/examples/store/main.go:89 +0x47
==================
==================
WARNING: DATA RACE
Write at 0x000004b4c8c0 by goroutine 82:
sync/atomic.AddInt64()
/usr/local/go/src/runtime/race_amd64.s:276 +0xb
github.com/go-ego/riot.(*Engine).indexerAddDocWorker()
/go/src/github.com/go-ego/riot/indexer_worker.go:52 +0x1a2
Previous read at 0x000004b4c8c0 by main goroutine:
github.com/go-ego/riot.(*Engine).FlushIndex()
/go/src/github.com/go-ego/riot/engine.go:656 +0x78
main.initEngine()
/go/src/github.com/go-ego/riot/examples/store/main.go:60 +0x378
github.com/go-ego/riot/vendor/github.com/go-ego/gse.(*Segmenter).LoadDict()
/go/src/github.com/go-ego/riot/vendor/github.com/go-ego/gse/segmenter.go:272 +0x35d
github.com/go-ego/riot.(*Engine).Init()
/go/src/github.com/go-ego/riot/engine.go:237 +0x9d2
main.initEngine()
/go/src/github.com/go-ego/riot/examples/store/main.go:34 +0x151
main.main()
/go/src/github.com/go-ego/riot/examples/store/main.go:89 +0x47
Goroutine 82 (running) created at:
github.com/go-ego/riot.(*Engine).Init()
/go/src/github.com/go-ego/riot/engine.go:275 +0x831
github.com/go-ego/riot/vendor/github.com/go-ego/gse.(*Segmenter).Read()
/go/src/github.com/go-ego/riot/vendor/github.com/go-ego/gse/segmenter.go:93 +0x64e
github.com/go-ego/riot/vendor/github.com/go-ego/gse.(*Segmenter).LoadDict()
/go/src/github.com/go-ego/riot/vendor/github.com/go-ego/gse/segmenter.go:272 +0x35d
github.com/go-ego/riot.(*Engine).Init()
/go/src/github.com/go-ego/riot/engine.go:237 +0x9d2
main.initEngine()
/go/src/github.com/go-ego/riot/examples/store/main.go:34 +0x151
main.main()
/go/src/github.com/go-ego/riot/examples/store/main.go:89 +0x47
==================
==================
WARNING: DATA RACE
Write at 0x000004b4c8c0 by goroutine 162:
sync/atomic.AddInt64()
/usr/local/go/src/runtime/race_amd64.s:276 +0xb
github.com/go-ego/riot.(*Engine).indexerAddDocWorker()
/go/src/github.com/go-ego/riot/indexer_worker.go:52 +0x1a2
Previous read at 0x000004b4c8c0 by main goroutine:
github.com/go-ego/riot.(*Engine).FlushIndex()
/go/src/github.com/go-ego/riot/engine.go:656 +0x78
main.initEngine()
/go/src/github.com/go-ego/riot/examples/store/main.go:60 +0x378
github.com/go-ego/riot/vendor/github.com/go-ego/gse.(*Segmenter).LoadDict()
/go/src/github.com/go-ego/riot/vendor/github.com/go-ego/gse/segmenter.go:272 +0x35d
github.com/go-ego/riot.(*Engine).Init()
/go/src/github.com/go-ego/riot/engine.go:237 +0x9d2
main.initEngine()
/go/src/github.com/go-ego/riot/examples/store/main.go:34 +0x151
main.main()
/go/src/github.com/go-ego/riot/examples/store/main.go:89 +0x47
Goroutine 162 (running) created at:
github.com/go-ego/riot.(*Engine).Init()
/go/src/github.com/go-ego/riot/engine.go:275 +0x831
github.com/go-ego/riot/vendor/github.com/go-ego/gse.(*Segmenter).Read()
/go/src/github.com/go-ego/riot/vendor/github.com/go-ego/gse/segmenter.go:93 +0x64e
github.com/go-ego/riot/vendor/github.com/go-ego/gse.(*Segmenter).LoadDict()
/go/src/github.com/go-ego/riot/vendor/github.com/go-ego/gse/segmenter.go:272 +0x35d
github.com/go-ego/riot.(*Engine).Init()
/go/src/github.com/go-ego/riot/engine.go:237 +0x9d2
main.initEngine()
/go/src/github.com/go-ego/riot/examples/store/main.go:34 +0x151
main.main()
/go/src/github.com/go-ego/riot/examples/store/main.go:89 +0x47
==================
==================
WARNING: DATA RACE
Write at 0x000004b4c8f8 by goroutine 309:
sync/atomic.AddInt64()
/usr/local/go/src/runtime/race_amd64.s:276 +0xb
github.com/go-ego/riot.(*Engine).storageIndexDocWorker()
/go/src/github.com/go-ego/riot/storage_worker.go:52 +0x43e
Previous read at 0x000004b4c8f8 by main goroutine:
github.com/go-ego/riot.(*Engine).FlushIndex()
/go/src/github.com/go-ego/riot/engine.go:658 +0x138
main.initEngine()
/go/src/github.com/go-ego/riot/examples/store/main.go:60 +0x378
github.com/go-ego/riot/vendor/github.com/go-ego/gse.(*Segmenter).LoadDict()
/go/src/github.com/go-ego/riot/vendor/github.com/go-ego/gse/segmenter.go:272 +0x35d
github.com/go-ego/riot.(*Engine).Init()
/go/src/github.com/go-ego/riot/engine.go:237 +0x9d2
main.initEngine()
/go/src/github.com/go-ego/riot/examples/store/main.go:34 +0x151
main.main()
/go/src/github.com/go-ego/riot/examples/store/main.go:89 +0x47
Goroutine 309 (running) created at:
github.com/go-ego/riot.(*Engine).Storage()
/go/src/github.com/go-ego/riot/engine.go:216 +0xbb3
github.com/go-ego/riot.(*Engine).Init()
/go/src/github.com/go-ego/riot/engine.go:289 +0x8df
github.com/go-ego/riot/vendor/github.com/go-ego/gse.(*Segmenter).Read()
/go/src/github.com/go-ego/riot/vendor/github.com/go-ego/gse/segmenter.go:93 +0x64e
github.com/go-ego/riot/vendor/github.com/go-ego/gse.(*Segmenter).LoadDict()
/go/src/github.com/go-ego/riot/vendor/github.com/go-ego/gse/segmenter.go:272 +0x35d
github.com/go-ego/riot.(*Engine).Init()
/go/src/github.com/go-ego/riot/engine.go:237 +0x9d2
main.initEngine()
/go/src/github.com/go-ego/riot/examples/store/main.go:34 +0x151
main.main()
/go/src/github.com/go-ego/riot/examples/store/main.go:89 +0x47
==================
==================
WARNING: DATA RACE
Write at 0x000004b4c8d0 by goroutine 42:
sync/atomic.AddInt64()
/usr/local/go/src/runtime/race_amd64.s:276 +0xb
github.com/go-ego/riot.(*Engine).indexerAddDocWorker()
/go/src/github.com/go-ego/riot/indexer_worker.go:55 +0x159
Previous read at 0x000004b4c8d0 by main goroutine:
github.com/go-ego/riot.(*Engine).FlushIndex()
/go/src/github.com/go-ego/riot/engine.go:667 +0x21a
main.initEngine()
/go/src/github.com/go-ego/riot/examples/store/main.go:60 +0x378
github.com/go-ego/riot/vendor/github.com/go-ego/gse.(*Segmenter).LoadDict()
/go/src/github.com/go-ego/riot/vendor/github.com/go-ego/gse/segmenter.go:272 +0x35d
github.com/go-ego/riot.(*Engine).Init()
/go/src/github.com/go-ego/riot/engine.go:237 +0x9d2
main.initEngine()
/go/src/github.com/go-ego/riot/examples/store/main.go:34 +0x151
main.main()
/go/src/github.com/go-ego/riot/examples/store/main.go:89 +0x47
Goroutine 42 (running) created at:
github.com/go-ego/riot.(*Engine).Init()
/go/src/github.com/go-ego/riot/engine.go:275 +0x831
github.com/go-ego/riot/vendor/github.com/go-ego/gse.(*Segmenter).Read()
/go/src/github.com/go-ego/riot/vendor/github.com/go-ego/gse/segmenter.go:93 +0x64e
github.com/go-ego/riot/vendor/github.com/go-ego/gse.(*Segmenter).LoadDict()
/go/src/github.com/go-ego/riot/vendor/github.com/go-ego/gse/segmenter.go:272 +0x35d
github.com/go-ego/riot.(*Engine).Init()
/go/src/github.com/go-ego/riot/engine.go:237 +0x9d2
main.initEngine()
/go/src/github.com/go-ego/riot/examples/store/main.go:34 +0x151
main.main()
/go/src/github.com/go-ego/riot/examples/store/main.go:89 +0x47
==================
==================
WARNING: DATA RACE
Write at 0x000004b4c8d0 by goroutine 22:
sync/atomic.AddInt64()
/usr/local/go/src/runtime/race_amd64.s:276 +0xb
github.com/go-ego/riot.(*Engine).indexerAddDocWorker()
/go/src/github.com/go-ego/riot/indexer_worker.go:55 +0x159
Previous read at 0x000004b4c8d0 by main goroutine:
github.com/go-ego/riot.(*Engine).FlushIndex()
/go/src/github.com/go-ego/riot/engine.go:667 +0x21a
main.initEngine()
/go/src/github.com/go-ego/riot/examples/store/main.go:60 +0x378
github.com/go-ego/riot/vendor/github.com/go-ego/gse.(*Segmenter).LoadDict()
/go/src/github.com/go-ego/riot/vendor/github.com/go-ego/gse/segmenter.go:272 +0x35d
github.com/go-ego/riot.(*Engine).Init()
/go/src/github.com/go-ego/riot/engine.go:237 +0x9d2
main.initEngine()
/go/src/github.com/go-ego/riot/examples/store/main.go:34 +0x151
main.main()
/go/src/github.com/go-ego/riot/examples/store/main.go:89 +0x47
Goroutine 22 (running) created at:
github.com/go-ego/riot.(*Engine).Init()
/go/src/github.com/go-ego/riot/engine.go:275 +0x831
github.com/go-ego/riot/vendor/github.com/go-ego/gse.(*Segmenter).Read()
/go/src/github.com/go-ego/riot/vendor/github.com/go-ego/gse/segmenter.go:93 +0x64e
github.com/go-ego/riot/vendor/github.com/go-ego/gse.(*Segmenter).LoadDict()
/go/src/github.com/go-ego/riot/vendor/github.com/go-ego/gse/segmenter.go:272 +0x35d
github.com/go-ego/riot.(*Engine).Init()
/go/src/github.com/go-ego/riot/engine.go:237 +0x9d2
main.initEngine()
/go/src/github.com/go-ego/riot/examples/store/main.go:34 +0x151
main.main()
/go/src/github.com/go-ego/riot/examples/store/main.go:89 +0x47
==================
==================
WARNING: DATA RACE
Write at 0x000004b4c8d0 by goroutine 122:
sync/atomic.AddInt64()
/usr/local/go/src/runtime/race_amd64.s:276 +0xb
github.com/go-ego/riot.(*Engine).indexerAddDocWorker()
/go/src/github.com/go-ego/riot/indexer_worker.go:55 +0x159
Previous read at 0x000004b4c8d0 by main goroutine:
github.com/go-ego/riot.(*Engine).FlushIndex()
/go/src/github.com/go-ego/riot/engine.go:667 +0x21a
main.initEngine()
/go/src/github.com/go-ego/riot/examples/store/main.go:60 +0x378
github.com/go-ego/riot/vendor/github.com/go-ego/gse.(*Segmenter).LoadDict()
/go/src/github.com/go-ego/riot/vendor/github.com/go-ego/gse/segmenter.go:272 +0x35d
github.com/go-ego/riot.(*Engine).Init()
/go/src/github.com/go-ego/riot/engine.go:237 +0x9d2
main.initEngine()
/go/src/github.com/go-ego/riot/examples/store/main.go:34 +0x151
main.main()
/go/src/github.com/go-ego/riot/examples/store/main.go:89 +0x47
Goroutine 122 (running) created at:
github.com/go-ego/riot.(*Engine).Init()
/go/src/github.com/go-ego/riot/engine.go:275 +0x831
github.com/go-ego/riot/vendor/github.com/go-ego/gse.(*Segmenter).Read()
/go/src/github.com/go-ego/riot/vendor/github.com/go-ego/gse/segmenter.go:93 +0x64e
github.com/go-ego/riot/vendor/github.com/go-ego/gse.(*Segmenter).LoadDict()
/go/src/github.com/go-ego/riot/vendor/github.com/go-ego/gse/segmenter.go:272 +0x35d
github.com/go-ego/riot.(*Engine).Init()
/go/src/github.com/go-ego/riot/engine.go:237 +0x9d2
main.initEngine()
/go/src/github.com/go-ego/riot/examples/store/main.go:34 +0x151
main.main()
/go/src/github.com/go-ego/riot/examples/store/main.go:89 +0x47
==================
==================
WARNING: DATA RACE
Write at 0x000004b4c8d0 by goroutine 82:
sync/atomic.AddInt64()
/usr/local/go/src/runtime/race_amd64.s:276 +0xb
github.com/go-ego/riot.(*Engine).indexerAddDocWorker()
/go/src/github.com/go-ego/riot/indexer_worker.go:55 +0x159
Previous read at 0x000004b4c8d0 by main goroutine:
github.com/go-ego/riot.(*Engine).FlushIndex()
/go/src/github.com/go-ego/riot/engine.go:667 +0x21a
main.initEngine()
/go/src/github.com/go-ego/riot/examples/store/main.go:60 +0x378
github.com/go-ego/riot/vendor/github.com/go-ego/gse.(*Segmenter).LoadDict()
/go/src/github.com/go-ego/riot/vendor/github.com/go-ego/gse/segmenter.go:272 +0x35d
github.com/go-ego/riot.(*Engine).Init()
/go/src/github.com/go-ego/riot/engine.go:237 +0x9d2
main.initEngine()
/go/src/github.com/go-ego/riot/examples/store/main.go:34 +0x151
main.main()
/go/src/github.com/go-ego/riot/examples/store/main.go:89 +0x47
Goroutine 82 (running) created at:
github.com/go-ego/riot.(*Engine).Init()
/go/src/github.com/go-ego/riot/engine.go:275 +0x831
github.com/go-ego/riot/vendor/github.com/go-ego/gse.(*Segmenter).Read()
/go/src/github.com/go-ego/riot/vendor/github.com/go-ego/gse/segmenter.go:93 +0x64e
github.com/go-ego/riot/vendor/github.com/go-ego/gse.(*Segmenter).LoadDict()
/go/src/github.com/go-ego/riot/vendor/github.com/go-ego/gse/segmenter.go:272 +0x35d
github.com/go-ego/riot.(*Engine).Init()
/go/src/github.com/go-ego/riot/engine.go:237 +0x9d2
main.initEngine()
/go/src/github.com/go-ego/riot/examples/store/main.go:34 +0x151
main.main()
/go/src/github.com/go-ego/riot/examples/store/main.go:89 +0x47
==================
==================
WARNING: DATA RACE
Write at 0x000004b4c8d0 by goroutine 102:
sync/atomic.AddInt64()
/usr/local/go/src/runtime/race_amd64.s:276 +0xb
github.com/go-ego/riot.(*Engine).indexerAddDocWorker()
/go/src/github.com/go-ego/riot/indexer_worker.go:55 +0x159
Previous read at 0x000004b4c8d0 by main goroutine:
github.com/go-ego/riot.(*Engine).FlushIndex()
/go/src/github.com/go-ego/riot/engine.go:667 +0x21a
main.initEngine()
/go/src/github.com/go-ego/riot/examples/store/main.go:60 +0x378
github.com/go-ego/riot/vendor/github.com/go-ego/gse.(*Segmenter).LoadDict()
/go/src/github.com/go-ego/riot/vendor/github.com/go-ego/gse/segmenter.go:272 +0x35d
github.com/go-ego/riot.(*Engine).Init()
/go/src/github.com/go-ego/riot/engine.go:237 +0x9d2
main.initEngine()
/go/src/github.com/go-ego/riot/examples/store/main.go:34 +0x151
main.main()
/go/src/github.com/go-ego/riot/examples/store/main.go:89 +0x47
Goroutine 102 (running) created at:
github.com/go-ego/riot.(*Engine).Init()
/go/src/github.com/go-ego/riot/engine.go:275 +0x831
github.com/go-ego/riot/vendor/github.com/go-ego/gse.(*Segmenter).Read()
/go/src/github.com/go-ego/riot/vendor/github.com/go-ego/gse/segmenter.go:93 +0x64e
github.com/go-ego/riot/vendor/github.com/go-ego/gse.(*Segmenter).LoadDict()
/go/src/github.com/go-ego/riot/vendor/github.com/go-ego/gse/segmenter.go:272 +0x35d
github.com/go-ego/riot.(*Engine).Init()
/go/src/github.com/go-ego/riot/engine.go:237 +0x9d2
main.initEngine()
/go/src/github.com/go-ego/riot/examples/store/main.go:34 +0x151
main.main()
/go/src/github.com/go-ego/riot/examples/store/main.go:89 +0x47
==================
2018/01/24 16:25:20 recover index number: 3
==================
WARNING: DATA RACE
Read at 0x00c420087f28 by goroutine 86:
github.com/go-ego/riot/core.(*Indexer).Lookup()
/go/src/github.com/go-ego/riot/core/indexer.go:333 +0xbc
github.com/go-ego/riot.(*Engine).indexerLookupWorker()
/go/src/github.com/go-ego/riot/indexer_worker.go:82 +0x8d8
Previous write at 0x00c420087f28 by goroutine 82:
github.com/go-ego/riot/core.(*Indexer).AddDocs()
/go/src/github.com/go-ego/riot/core/indexer.go:220 +0x133f
github.com/go-ego/riot/core.(*Indexer).AddDocToCache()
/go/src/github.com/go-ego/riot/core/indexer.go:147 +0x8bc
github.com/go-ego/riot.(*Engine).indexerAddDocWorker()
/go/src/github.com/go-ego/riot/indexer_worker.go:48 +0x127
Goroutine 86 (running) created at:
github.com/go-ego/riot.(*Engine).Init()
/go/src/github.com/go-ego/riot/engine.go:281 +0x774
github.com/go-ego/riot/vendor/github.com/go-ego/gse.(*Segmenter).Read()
/go/src/github.com/go-ego/riot/vendor/github.com/go-ego/gse/segmenter.go:93 +0x64e
github.com/go-ego/riot/vendor/github.com/go-ego/gse.(*Segmenter).LoadDict()
/go/src/github.com/go-ego/riot/vendor/github.com/go-ego/gse/segmenter.go:272 +0x35d
github.com/go-ego/riot.(*Engine).Init()
/go/src/github.com/go-ego/riot/engine.go:237 +0x9d2
main.initEngine()
/go/src/github.com/go-ego/riot/examples/store/main.go:34 +0x151
main.main()
/go/src/github.com/go-ego/riot/examples/store/main.go:89 +0x47
Goroutine 82 (running) created at:
github.com/go-ego/riot.(*Engine).Init()
/go/src/github.com/go-ego/riot/engine.go:275 +0x831
github.com/go-ego/riot/vendor/github.com/go-ego/gse.(*Segmenter).Read()
/go/src/github.com/go-ego/riot/vendor/github.com/go-ego/gse/segmenter.go:93 +0x64e
github.com/go-ego/riot/vendor/github.com/go-ego/gse.(*Segmenter).LoadDict()
/go/src/github.com/go-ego/riot/vendor/github.com/go-ego/gse/segmenter.go:272 +0x35d
github.com/go-ego/riot.(*Engine).Init()
/go/src/github.com/go-ego/riot/engine.go:237 +0x9d2
main.initEngine()
/go/src/github.com/go-ego/riot/examples/store/main.go:34 +0x151
main.main()
/go/src/github.com/go-ego/riot/examples/store/main.go:89 +0x47
==================
==================
WARNING: DATA RACE
Read at 0x00c420088268 by goroutine 166:
github.com/go-ego/riot/core.(*Indexer).Lookup()
/go/src/github.com/go-ego/riot/core/indexer.go:333 +0xbc
github.com/go-ego/riot.(*Engine).indexerLookupWorker()
/go/src/github.com/go-ego/riot/indexer_worker.go:82 +0x8d8
Previous write at 0x00c420088268 by goroutine 162:
github.com/go-ego/riot/core.(*Indexer).AddDocs()
/go/src/github.com/go-ego/riot/core/indexer.go:220 +0x133f
github.com/go-ego/riot/core.(*Indexer).AddDocToCache()
/go/src/github.com/go-ego/riot/core/indexer.go:147 +0x8bc
github.com/go-ego/riot.(*Engine).indexerAddDocWorker()
/go/src/github.com/go-ego/riot/indexer_worker.go:48 +0x127
Goroutine 166 (running) created at:
github.com/go-ego/riot.(*Engine).Init()
/go/src/github.com/go-ego/riot/engine.go:281 +0x774
github.com/go-ego/riot/vendor/github.com/go-ego/gse.(*Segmenter).Read()
/go/src/github.com/go-ego/riot/vendor/github.com/go-ego/gse/segmenter.go:93 +0x64e
github.com/go-ego/riot/vendor/github.com/go-ego/gse.(*Segmenter).LoadDict()
/go/src/github.com/go-ego/riot/vendor/github.com/go-ego/gse/segmenter.go:272 +0x35d
github.com/go-ego/riot.(*Engine).Init()
/go/src/github.com/go-ego/riot/engine.go:237 +0x9d2
main.initEngine()
/go/src/github.com/go-ego/riot/examples/store/main.go:34 +0x151
main.main()
/go/src/github.com/go-ego/riot/examples/store/main.go:89 +0x47
Goroutine 162 (running) created at:
github.com/go-ego/riot.(*Engine).Init()
/go/src/github.com/go-ego/riot/engine.go:275 +0x831
github.com/go-ego/riot/vendor/github.com/go-ego/gse.(*Segmenter).Read()
/go/src/github.com/go-ego/riot/vendor/github.com/go-ego/gse/segmenter.go:93 +0x64e
github.com/go-ego/riot/vendor/github.com/go-ego/gse.(*Segmenter).LoadDict()
/go/src/github.com/go-ego/riot/vendor/github.com/go-ego/gse/segmenter.go:272 +0x35d
github.com/go-ego/riot.(*Engine).Init()
/go/src/github.com/go-ego/riot/engine.go:237 +0x9d2
main.initEngine()
/go/src/github.com/go-ego/riot/examples/store/main.go:34 +0x151
main.main()
/go/src/github.com/go-ego/riot/examples/store/main.go:89 +0x47
==================
search--------- {[google testing] [{3 Google is testing another Search results layout with
rounded cards, new colors, and the 4 mysterious colored dots again <nil> <nil> [0] [] []}] false 1} ; docs= [{3 Google is testing another Search results layout with
rounded cards, new colors, and the 4 mysterious colored dots again <nil> <nil> [0] [] []}]
Found 11 data race(s)Like this:
searcher.Search(types.SearchReq{
TextMayBeEmpty: true,
Text: "",
RankOpts: &types.RankOpts{
OutputOffset: 0,
MaxOutputs: 100,
},
})It's useful for providing web api
// at `engine.go:775 and 791`
docsId = append(docsId, uint64(k[0]))unint64(k[0]) only convert the first byte of k to uint64, not the full docId.
...
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.