Code Monkey home page Code Monkey logo

Comments (7)

timshannon avatar timshannon commented on July 3, 2024

40-70 ms is an eternity. To start troubleshooting the issue the first thing I'd try is removing bolthold from the equation entirely. Just do inserts directly into boltdb, and see if it's still taking that long.

from bolthold.

petermakeswebsites avatar petermakeswebsites commented on July 3, 2024

I actually switched to the sister version - badgerhold - and it went down to 20ms... Still crazy long IMO. How much time is considered normal in your experience for a single write? Just so I have a vague target for my debugging. I'll try doing it directly to Bolt instead and see what happens

from bolthold.

timshannon avatar timshannon commented on July 3, 2024

I'll throw together some quick timings on my machine, which has spinning metal and ssd's.

BadgerDB's storage format is designed specifically to take advantage of SSD's where as Bolt doesn't.

https://dgraph.io/blog/post/badger/

Looking at dgraph's benchmarks you should see inserts at around 9ms depending on the size of the insert.

And if there are index inserts, then you should double that number 9ms for the main insert, and another 9 ms for the index insert.

from bolthold.

petermakeswebsites avatar petermakeswebsites commented on July 3, 2024

So I tried to access bolt directly via boltholdstore.db.Bolt()...

All I can say is wow. I wrote my own pretty straightforward put transaction with a bucket NextSequence. The tx.commit part of the tx alone takes ~60ms. That's just crazy. I was working on my own database previously and was getting about 70 us (0.07ms) per write with Badger. Granted, badger is faster than Bolt... but... a thousand times faster is definitely fishy. No clue what could be slowing it down but I'll continue to poke around.

It seems at this point that this isn't a bolthold thing at least. Though, it's still possible that some init setting is causing this slow write speed. Not sure.

I'll do another test setting up my own bolt db instead of accessing it through bolthold and see if it makes a difference. Can't imagine it will, but then at least then I can totally rule out bolthold.

from bolthold.

timshannon avatar timshannon commented on July 3, 2024

Thanks for relaying the info. I know boltdb and badger have very different goals, where as bolt aims for simplicity and (https://github.com/boltdb/bolt#leveldb-rocksdb) and badger is designed for with a file format catered towards SSDs.

from bolthold.

HeavyHorst avatar HeavyHorst commented on July 3, 2024

I was working on my own database previously and was getting about 70 us (0.07ms) per write with Badger.

BadgerDB doesn't sync writes with the default config as far as i know.

  package main                                                                                                                                                                                                       
                                                                                                                                                                                                                     
  import (                                                                                                                                                                                                           
      "fmt"                                                                                                                                                                                                          
      "time"                                                                                                                                                                                                         
                                                                                                                                                                                                                     
      "github.com/timshannon/badgerhold"                                                                                                                                                                             
  )                                                                                                                                                                                                                  
                                                                                                                                                                                                                     
  type PostType struct {                                                                                                                                                                                             
      Name       string   `json:"name"`                                                                                                                                                                              
      PluralName string   `json:"pluralName"`                                                                                                                                                                        
      PostTypeID uint64   `json:"postTypeID" boltholdKey:"PostTypeID"`                                                                                                                                               
      Fields     FieldMap `json:"fields"`                                                                                                                                                                            
  }                                                                                                                                                                                                                  
                                                                                                                                                                                                                     
  type FieldMap map[uint64]Field                                                                                                                                                                                     
                                                                                                                                                                                                                     
  type Field struct {                                                                                                                                                                                                
      Name    string                                                                                                                                                                                                 
      Kind    string                                                                                                                                                                                                 
      Slug    string                                                                                                                                                                                                 
      Options string                                                                                                                                                                                                 
  }                                                                                                                                                                                                                  
                                                                                                                                                                                                                     
  func main() {                                                                                                                                                                                                      
      options := badgerhold.DefaultOptions                                                                                                                                                                           
      options.Dir = "data"                                                                                                                                                                                           
      options.ValueDir = "data"                                                                                                                                                                                      
      options.SyncWrites = false                                                                                                                                                                                     
      d, err := badgerhold.Open(options)                                                                                                                                                                             
      if err != nil {                                                                                                                                                                                                
          panic(err)                                                                                                                                                                                                 
      }                                                                                                                                                                                                              
                                                                                                                                                                                                                     
      pt := PostType{}                                                                                                                                                                                               
                                                                                                                                                                                                                     
      for i := 0; i < 100; i++ {                                                                                                                                                                                     
          x := time.Now()                                                                                                                                                                                            
          err = d.Insert(badgerhold.NextSequence(), &pt)                                                                                                                                                             
          fmt.Println(err)                                                                                                                                                                                           
          fmt.Println("Creation time: ", time.Since(x))                                                                                                                                                              
      }                                                                                                                                                                                                              
  }

With this config i get (badgerhold):
Creation time: 154.381µs
Creation time: 108.543µs
Creation time: 140.977µs
Creation time: 120.703µs
...

With activated sync writes and badgerhold (saver):
Creation time: 13.996905ms
Creation time: 24.53465ms
Creation time: 10.498898ms
Creation time: 18.433725ms
....

With Bolthold:
Creation time: 32.025662ms
Creation time: 46.942041ms
Creation time: 45.16791ms
Creation time: 37.354526ms
...

With bolthold and NoSync flag=true:
Creation time: 151.732µs
Creation time: 147.138µs
Creation time: 114.893µs
Creation time: 132.968µs
...

This is on a relatively slow pc:
Intel(R) Xeon(R) CPU X5650 @ 2.67GHz
Samsung SSD 840 Series SATA 120GB SSD

from bolthold.

HeavyHorst avatar HeavyHorst commented on July 3, 2024

Same test on my Notebook:
Intel(R) Core(TM) i7-7500U CPU @ 2.70GHz
SanDisk SD8SN8U512G1002 512GB M2 SSD

With this config i get (badgerhold):
Creation time: 39.432µs
Creation time: 38.369µs
Creation time: 45.024µs
Creation time: 54.537µs

With activated sync writes and badgerhold (saver):
Creation time: 1.762754ms
Creation time: 1.832461ms
Creation time: 1.720504ms
Creation time: 1.809905ms

With Bolthold:
Creation time: 1.632288ms
Creation time: 1.702195ms
Creation time: 1.65075ms
Creation time: 1.685826ms

With bolthold and NoSync flag=true:
Creation time: 70.892µs
Creation time: 67.832µs
Creation time: 76.844µs
Creation time: 96.243µs

from bolthold.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.