Performance insanity! about atomspace-rocks HOT 3 CLOSED

FYI, re-opening the same dataset, and storing exactly the same data back again, gives an insanely different result:

...
Stored 440000 of 510993 left-wilds in 3 secs (13333 pairs/sec)
Stored 480000 of 510993 left-wilds in 4 secs (10000 pairs/sec)
Done storing 510993 left-wilds in 43 secs

Wow!? This suggests that the first time, the Rocks bottleneck was that, perhaps, the indexes were being recomputed with each store!?

from atomspace-rocks.

This behavior appears to have been fixed in eedfbc7 ... That commit redesigned the incoming set layout....

The store of left-wilds means that the wild-cards have thousands or more incoming links. The old design required thousands of read-modify-write edits, which cratered rocks with vast numbers of bloated *.sst files. The redesign avoids this.

Basically, the old design was editing a multi-kbyte string of incoming sets, and storing a new string hundreds of times a second. This spewed megabytes per second of modified values into rocks, a gigabyte or two of modifications every minute, and the log-structured design of rocks could not keep up with the rapid pace of edits. The new design creates a brand new key for each elt of the incoming set. This makes atomic updates faster, simpler. So .. win-win.

from atomspace-rocks.

Closing. Yesterday's run shows:

Stored 80000 of 485786 left-wilds in 14 secs (2857 pairs/sec)        
Stored 120000 of 485786 left-wilds in 16 secs (2500 pairs/sec)     
Stored 160000 of 485786 left-wilds in 16 secs (2500 pairs/sec)    
Stored 200000 of 485786 left-wilds in 14 secs (2857 pairs/sec)    
Stored 240000 of 485786 left-wilds in 16 secs (2500 pairs/sec)    
Stored 280000 of 485786 left-wilds in 17 secs (2353 pairs/sec)  
Stored 320000 of 485786 left-wilds in 18 secs (2222 pairs/sec)   
Stored 360000 of 485786 left-wilds in 17 secs (2353 pairs/sec)  
Stored 400000 of 485786 left-wilds in 14 secs (2857 pairs/sec)                  
Stored 440000 of 485786 left-wilds in 15 secs (2667 pairs/sec)                  
Stored 480000 of 485786 left-wilds in 15 secs (2667 pairs/sec)
Done storing 485786 left-wilds in 189 secs

This is a vast improvement; it is now 2246 / 189 = 12x faster than postgres! Yay!!

from atomspace-rocks.