Comments (7)
40-70 ms is an eternity. To start troubleshooting the issue the first thing I'd try is removing bolthold from the equation entirely. Just do inserts directly into boltdb, and see if it's still taking that long.
from bolthold.
I actually switched to the sister version - badgerhold - and it went down to 20ms... Still crazy long IMO. How much time is considered normal in your experience for a single write? Just so I have a vague target for my debugging. I'll try doing it directly to Bolt instead and see what happens
from bolthold.
I'll throw together some quick timings on my machine, which has spinning metal and ssd's.
BadgerDB's storage format is designed specifically to take advantage of SSD's where as Bolt doesn't.
https://dgraph.io/blog/post/badger/
Looking at dgraph's benchmarks you should see inserts at around 9ms depending on the size of the insert.
And if there are index inserts, then you should double that number 9ms for the main insert, and another 9 ms for the index insert.
from bolthold.
So I tried to access bolt directly via boltholdstore.db.Bolt()...
All I can say is wow. I wrote my own pretty straightforward put transaction with a bucket NextSequence. The tx.commit part of the tx alone takes ~60ms. That's just crazy. I was working on my own database previously and was getting about 70 us (0.07ms) per write with Badger. Granted, badger is faster than Bolt... but... a thousand times faster is definitely fishy. No clue what could be slowing it down but I'll continue to poke around.
It seems at this point that this isn't a bolthold thing at least. Though, it's still possible that some init setting is causing this slow write speed. Not sure.
I'll do another test setting up my own bolt db instead of accessing it through bolthold and see if it makes a difference. Can't imagine it will, but then at least then I can totally rule out bolthold.
from bolthold.
Thanks for relaying the info. I know boltdb and badger have very different goals, where as bolt aims for simplicity and (https://github.com/boltdb/bolt#leveldb-rocksdb) and badger is designed for with a file format catered towards SSDs.
from bolthold.
I was working on my own database previously and was getting about 70 us (0.07ms) per write with Badger.
BadgerDB doesn't sync writes with the default config as far as i know.
package main
import (
"fmt"
"time"
"github.com/timshannon/badgerhold"
)
type PostType struct {
Name string `json:"name"`
PluralName string `json:"pluralName"`
PostTypeID uint64 `json:"postTypeID" boltholdKey:"PostTypeID"`
Fields FieldMap `json:"fields"`
}
type FieldMap map[uint64]Field
type Field struct {
Name string
Kind string
Slug string
Options string
}
func main() {
options := badgerhold.DefaultOptions
options.Dir = "data"
options.ValueDir = "data"
options.SyncWrites = false
d, err := badgerhold.Open(options)
if err != nil {
panic(err)
}
pt := PostType{}
for i := 0; i < 100; i++ {
x := time.Now()
err = d.Insert(badgerhold.NextSequence(), &pt)
fmt.Println(err)
fmt.Println("Creation time: ", time.Since(x))
}
}
With this config i get (badgerhold):
Creation time: 154.381µs
Creation time: 108.543µs
Creation time: 140.977µs
Creation time: 120.703µs
...
With activated sync writes and badgerhold (saver):
Creation time: 13.996905ms
Creation time: 24.53465ms
Creation time: 10.498898ms
Creation time: 18.433725ms
....
With Bolthold:
Creation time: 32.025662ms
Creation time: 46.942041ms
Creation time: 45.16791ms
Creation time: 37.354526ms
...
With bolthold and NoSync flag=true:
Creation time: 151.732µs
Creation time: 147.138µs
Creation time: 114.893µs
Creation time: 132.968µs
...
This is on a relatively slow pc:
Intel(R) Xeon(R) CPU X5650 @ 2.67GHz
Samsung SSD 840 Series SATA 120GB SSD
from bolthold.
Same test on my Notebook:
Intel(R) Core(TM) i7-7500U CPU @ 2.70GHz
SanDisk SD8SN8U512G1002 512GB M2 SSD
With this config i get (badgerhold):
Creation time: 39.432µs
Creation time: 38.369µs
Creation time: 45.024µs
Creation time: 54.537µs
With activated sync writes and badgerhold (saver):
Creation time: 1.762754ms
Creation time: 1.832461ms
Creation time: 1.720504ms
Creation time: 1.809905ms
With Bolthold:
Creation time: 1.632288ms
Creation time: 1.702195ms
Creation time: 1.65075ms
Creation time: 1.685826ms
With bolthold and NoSync flag=true:
Creation time: 70.892µs
Creation time: 67.832µs
Creation time: 76.844µs
Creation time: 96.243µs
from bolthold.
Related Issues (20)
- 如何查询某个bucket的全部数据 HOT 1
- his error occurs when fetching data from a specific bucket HOT 8
- readme.md: boltholdIndex, boltHoldIndex HOT 4
- Upsert with custom Storer delivers inconsistent value type to index function HOT 2
- Multiple calls to Delete for non-existing key result in EOF error instead of ErrNotFound HOT 2
- Add method to wrap existing bolt instance HOT 1
- Q: limit query without conditions HOT 2
- Unique index
- Can't sort by PK HOT 1
- Add support for batch inserts HOT 2
- Distinct constraint HOT 2
- Get vs FindOne HOT 9
- Web UI
- Aggregate query on nested struct HOT 2
- Constraint pairs HOT 1
- Initialise read only store HOT 2
- foreach can't use, when the key's type is uint64 HOT 1
- SliceIndex query ContainsAll not working HOT 6
- Sort by primary key has no effect seemingly HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bolthold.