Here's some weird unexpected, confusing behavior w.r.t. frames. It's both a bug and a user-error. It is exhibited by the following:

(use-modules (opencog) (opencog exec))
(use-modules (opencog persist))
(use-modules (opencog persist-rocks))

(define a (AtomSpace))
(define b (AtomSpace))
(define c (AtomSpace a b))
(cog-set-atomspace! a)
(Concept "foo")
(Concept "bar")
(Concept "baz")
(Concept "I'm in A")

(cog-set-atomspace! c)
(define rsn (RocksStorageNode "rocks:///tmp/foo"))
(cog-open rsn)

; Oh no! Saving atomspace contents without saving frames first!!
(store-atomspace)
(cog-rocks-print rsn "")

; Now store atomspace b (which should be empty!!)
(store-frames b)

; Hmm looks like all atoms got assigned to atomspace b! Oh no!
(cog-rocks-print rsn "")

If the store-frames is done first, before the store-atomspace, then the correct behavior is seen. This is a user-error, since the user should have done the store-frames first. It is also a bug, since the store-frames should not have re-assigned the atoms to the wrong atomspace.

In the context of #15 , while testing the Multiframe-Example
following happened:
When I tried to connect the union atomspace to a RocksDB and then try cog-prt-atomspace, I got a segfault.:

(cog-set-atomspace! c)
(define rsn (RocksStorageNode "rocks:///home/opcwdir/repodock/ForTestingStuff/foo.rdb"))
(cog-open rsn)
(store-atomspace)
(cog-prt-atomspace)

This happens also, if I use cog-rocks-open.
Printing the content with cog-rocks-print works fine.

Not sure, if this is a realistic use case.

On small datasets and on synthetic benchmarks, the RocksDB backend is 2x or 3x faster than the Postgres backend. This is not surprising, given the complexity of mapping the AtomSpace to SQL, together with the overhead of client-server communications. However, in a real-world use-case, RocksDB is 9x slower than Postgres. Why? What is going wrong? Is this related to issues #10 and #9? Will fixes to those also fix this?

From the learning project. RocksDB:

(define wsv (make-shape-vec-api))
(define wss (add-pair-stars wsv))
(define wst (batch-transpose wss))
(wst 'mmt-marginals)
...
Stored 40000 of 510993 left-wilds in 101 secs (396 pairs/sec)
Stored 80000 of 510993 left-wilds in 356 secs (112 pairs/sec)
Stored 120000 of 510993 left-wilds in 607 secs (66 pairs/sec)
Stored 160000 of 510993 left-wilds in 865 secs (46 pairs/sec)
Stored 200000 of 510993 left-wilds in 1075 secs (37 pairs/sec)
Stored 240000 of 510993 left-wilds in 1352 secs (30 pairs/sec)
Stored 280000 of 510993 left-wilds in 1591 secs (25 pairs/sec)
Stored 320000 of 510993 left-wilds in 1841 secs (22 pairs/sec)
Stored 360000 of 510993 left-wilds in 2076 secs (19 pairs/sec)
Stored 400000 of 510993 left-wilds in 2296 secs (17 pairs/sec)
Stored 440000 of 510993 left-wilds in 2584 secs (15 pairs/sec)
Stored 480000 of 510993 left-wilds in 2871 secs (14 pairs/sec)
Done storing 510993 left-wilds in 19969 secs

Freakin disaster. That works out to 5.5 hours, or 26 pairs/second. Compare this to exactly the same dataset on Postgres:

Stored 40000 of 510993 left-wilds in 170 secs (235 pairs/sec)
Stored 80000 of 510993 left-wilds in 168 secs (238 pairs/sec)
Stored 120000 of 510993 left-wilds in 188 secs (213 pairs/sec)
Stored 160000 of 510993 left-wilds in 177 secs (226 pairs/sec)
Stored 200000 of 510993 left-wilds in 171 secs (234 pairs/sec)
Stored 240000 of 510993 left-wilds in 165 secs (242 pairs/sec)
Stored 280000 of 510993 left-wilds in 164 secs (244 pairs/sec)
Stored 320000 of 510993 left-wilds in 176 secs (227 pairs/sec)
Stored 360000 of 510993 left-wilds in 163 secs (245 pairs/sec)
Stored 400000 of 510993 left-wilds in 189 secs (212 pairs/sec)
Stored 440000 of 510993 left-wilds in 187 secs (214 pairs/sec)
Stored 480000 of 510993 left-wilds in 185 secs (216 pairs/sec)
Done storing 510993 left-wilds in 2246 secs

Postgres took 37 minutes to store the same dataset. What's the problem, here? How can we fix this?

The (cog-node TYPE NAME) function allows a user to ask "does a node with the given TYPE and NAME exist in the AtomSpace?", without actually creating that Node. A similar function is needed for the RocksStorageNode: to ask if an Atom is being held, without actually creating it.

This needs modifications the generic StorageNode API, so that this question can be asked w/ the generic API, which then trickles down to the specific StorageNode.

In the same vein, other useful functions would be modeled on (cog-incoming-size ATOM) and (cog-incoming-size-by-type ATOM TYPE) to report on the incoming set of an ATOM held in storage, without actually fetching that incoming set.

The (cog-report-counts) function provides a summary of the Atom types stored in the AtomSpace. A similar function is needed, to report what is held in a RocksStorageNode (without actually loading those atoms, of course).

The simple version of this would be to modify the existing report function to total these up and print them. The fancy version of this would be to also add the API to the generic StorageNode API, to make it usable by all backends.

See WiscKey
and the corresponding RocksDB plugin: tikv/titan

Running the learning kit on a tiny dictionary resulted in this:

$ du -s *rdb
21710584        gram-2.rdb   <<<<<<<<<<< wtf 21GB really?

closing and opening gives:

538360  gram-2.rdb  << half a GB. That's more like it.

which is a 40x compaction. That's sick. That's too big.

When multiple frames are being used, the current rocks code will just hide atoms, instead of removing them, when frames are being used. This results in functionally correct behavior, but is wasteful of storage if the atom can actually be deleted.

The AtomSpace extract code currently implements the correct (reference) implementation: it will either hide an atom, when needed, or it will delete the atom, if possible. It can be used as the final arbiter of whether to delete, or not. The backend should follow this advice.

There right way to solve this is to implement the pre-delete and a post-delete calls in the backend. The atomspace calls pre-delete before doing the deletion, and rocks can gather any needed info for the deletion to happen. Next, the atomspace extracts the atom. Then it calls the post-delete hook. The hook code should look to see what the atomspace did: either the absent flag is set on the atom, or the atom is actually gone. If its only marked absent, then rocks should also hide the atom. else, rocks should delete the atom.

During learning on a tiny grammar, RAM usage by RocksDB exploded to 90GBytes. This is .. insane, it should not be more than a few GBytes for this workload. This is 40x greater RAM usage than expected ... the 40x number is just like the one in issue #9 and might be curable in the same way...

Before (cog-atomspace-clear), the cog-rocks-print works as expected.
If I try to inspect the RocksDB after (cog-atomspace-clear), I get:

scheme@(guile-user)> (cog-rocks-print rsn  "a@")
ice-9/boot-9.scm:1685:16: In procedure raise-exception:
In procedure cog-rocks-print: Wrong type (expecting opencog atom): #<Invalid handle>

Entering a new prompt.  Type `,bt' for a backtrace or `,q' to continue.
scheme@(guile-user) [1]> 

As I understand it, rocks should not be affected if you clear the AS in RAM.

Per discussion in opencog/atomspace#2787 (comment) if a RocksDB database holds an atom type that the C++ code does not know about, then it should be auto-created in the atomspace. To make this possible, the RocksDB tables will need to store the atom inheritance hierarchy, which they do not currently do.

Perhaps this could be best achieved by formally defining new atom types in atomese!? (and then just storing the atomese!?)

See also opencog/atomspace#2789

s-expressions are OK, but the database could be smaller if they were encoded by e.g. using an integer instead of the spelled-out type-name. Additional compression is possible, but at some point, the cpu-time of additional compression/decompression outweighs storage savings. Note also RocksDB has built-in compression, so this idea is kind-of questionable anyway?

Two alpha-equivalent Atoms will be stored s distinct entries. This will result in duplicate, corrupted Values associated with the alpha-equivalent Atoms.

The fix for this seems to involve storing the 64-bit hash and using that to detect possible alpha-equivalence. All Atoms with the same hash would have to be fetched, and then tested for possible alpha-equivalence. The getLink() and getNode() callbacks are problematic, since they don't come with a pre-computed hash, and so have to pay the extra cost of computing it.

FWIW, this problem might affect other back-ends too, e.g. the sql backend, and maybe the cogserver backend? This problem has not been explored...