Well first off all, I like it very much. The storage engine is nice and well documented. I like how it's follows physical storage model, but dynamic size allocation. I wish I could get it for python lol.
Anyway trying to see on how I can get a key if I supply just a data. The swagger looks, like it doesn't return anything but 200 when you store. Is there special sequence provider I overlooked?
Thirdly, partitions are nice, but I liked how OrientDb linked them to the types(classes) so if you store Line db knows which partition it goes to. Makes for explicit configuration, but more fluid application coding. Just a suggestion.
Fourthly, do you hang on Gitter or any other channels to talk about this project other then issues section?

Hi! This looks like an interesting project, just wanted to make sure it's still active before I start playing with it?

How fast are lookups? What is a complexity?

We can create Certs that are trusted in all browsers for local development.

It will make development easier i think and lead to less false positives like:

2021/05/17 13:29:26 http: TLS handshake error from [::1]:50088: remote error: tls: unknown certificate

For Dev environment ...

https://github.com/FiloSottile/mkcert

It is golang and so can be imported or just used independently.

I just hit a problem with running tutorial from the examples dir ( not dist) to do with TLS trust.
mkcert is nice because the many browsers trust the dev server, and so you don't get any false positives and other weird stuff.

http: TLS handshake error from [::1]:63645: remote error: tls: unknown certificate

As an aside, Its important to also turn this off for Prod, since normally the Proxy Server or whatever you use will do this for you.

For Prod environment ...

It might be useful to add LetEncrypt that caddy uses ?

I see two options here:

1. Get use Caddy, and replace NGinx. The caddy config will then do it all for you.
2. Use https://github.com/caddyserver/certmagic imported into eliasdb itself. You will need to make this configurable because if your using a Proxy, the proxy ( like Caddy or Nginy or whatever) will do all this for you.

The durable storage of the Certs is also a consideration, because you want all instances in the cluster to share the same Cert.
More info here: https://caddy.community/t/where-does-caddy-keep-their-certificates/4728
Same goes for local Dev too, in that you want the Certs stored globally for all instances.

Hello,
I'm digging into eliasdb for few days and I'm stuck with a 'query where clause' with strings.
I want to query nodes with a date string greater than a given string :
get message where date > 2017-03-21 for example. But I get the error Value of operand is not a number.

Does this mean that it is not possible to make strings comparaison in EQL ?
Thanks in advance for your response.

Looks like a great project! We need more embeddable storage engines for Go.

1. Just want to point out that the way code is structured is Java-like. Idiomatic Go code doesn't nest code (i.e. nosrc/devt.de).

The idea is to have import paths as short as possible i.e. instead of import "github.com/kroti/eliasdb/src/dev.de/eliasdb" I should just be able to import "github.com/kroti/eliasdb".

If you consider embeddable library to be main artifact, then github.com/kroti/eilasdb would be an import path for the library.

Executables that are part of the repo and use the library would go into cmd directory (e.g. the server would go into cmd/server).

You can see this structure in both https://github.com/boltdb/bolt and https://github.com/golang/leveldb.

The code can be divided into sub-packages, but from the perspective of API design it would be best if code using the library could do most of the work with just API exposed by top-level API.

In general idiomatic Go code favors "fatter" packages than idiomatic Java code. You can see that boltdb code doesn't even have sub-packages even though it could be logically divided into smaller parts. Another example is https://github.com/cznic/ql (it's embeddable sql database which pretty much uses only top-level package for all the code).

See also https://blog.golang.org/organizing-go-code

2. relative imports are frown upon in Go i.e. instead of import "devt.de/eliasdb/storage" it should be fully qualified `import "github.com/krotik/eliasdb/devt.de/eliasdb/storage". Using "devt.de/..." only works if you change GOPATH which breaks standard Go workflow i.e. the package is not usably for 99% of Go programmers. This is explicitly mentioned as a thing not to do in https://blog.golang.org/organizing-go-code

3. Idiomatic Go code discourages "stuttering" i.e. repeating package name in names of exported functions, types etc. in that package.

For example in package graphstorage, type GraphStorage interface repeats "graph", so it would be type Storage interface. See, for example, how in https://github.com/golang/leveldb/blob/master/record/record.go there is type Reader and not type RecordReader, because even though Reader would be ambiguous globally, it's not ambiguous when scoped to reader package and in Go everything is fully qualified.

See e.g. https://talks.golang.org/2014/names.slide#13

Hello,

I am in search of a good Graph DB in Go that can be used as part of a huge P2P project in which there could be a large number of nodes dynamically joining and leaving at various times.

Can you please tell me how well Eliasdb scales?
Thanks in advance

Very cool project! Just saw it today for the first time.

What are your thoughts on supporting nested JSON structures rather than just primitives within nodes? This is a requirement for the project I'm working on, and I'm willing to help if it's a feature you'd care to add.

For example, earlier today after working through the tutorial, I tried to store the JSON structure below:

> store
{
   "nodes": [
      {
         "key":"mytest",
         "kind":"Test",
         "int":42,
         "float":3.1415926,
         "str":"foo bar",
         "nested":{
            "nested_int":12,
            "nested_float":1.234,
            "nested_str":"time flies like an arrow"
         }
      }
   ]
}

The first try, I got this result:

SyntaxError: JSON.parse: expected property name or '}' at line 1 column 4 of the JSON data

After validating the JSON, I suspected that might be masking a different error, so I put all of the JSON on a single line and tried again:

GraphError: Could not write graph information (gob: type not registered for interface: map[string]interface {}) 

So I added the appropriate gob.Register call, and rebuilt. After doing this, the store operation succeeded, but I was of course unable to query based on nested values. With this feature, I would expect the node to be returned by all the following queries:

> get Test where nested.nested_int = 12
> get Test where nested.nested_float > 1.0
> get Test where nested.nested_str beginswith "time"
> index Test nested.nested_int value 12
> index Test nested.nested_str word "flies"

this code:
https://github.com/krotik/eliasdb/tree/master/examples/tutorial

env

am on mac if it helps :)
golang 1.6.3

I am running this from dist directory BTW.

Terminal

it works 👍

>>>get Line
┌─────────┬────────────────────────┐
│Line Key │Line Name               │
│1:n:key  │1:n:name                │
├─────────┼────────────────────────┤
│10       │Piccadilly Line         │
│13       │Docklands Light Railway │
│12       │Waterloo & City Line    │
│11       │Victoria Line           │
│8        │Metropolitan Line       │
│1        │Bakerloo Line           │
│2        │Central Line            │
│3        │Circle Line             │
│9        │Northern Line           │
│5        │East London Line        │
│7        │Jubilee Line            │
│6        │Hammersmith & City Line │
│4        │District Line           │
└─────────┴────────────────────────┘
>>>

find liverpool
Partition main
Kind      Station
┌─────────┬────┬────────┬─────────┬──────────┬─────────────────┬─────┐
│has_rail │key │kind    │latitude │longitude │name             │zone │
├─────────┼────┼────────┼─────────┼──────────┼─────────────────┼─────┤
│true     │156 │Station │51.5178  │-0.0823   │Liverpool Street │1    │
└─────────┴────┴────────┴─────────┴──────────┴─────────────────┴─────┘

Term browser

same query fails 👎

I am looking for Go db that supports spatial data and I found only DGraph but after an hour of not being able to figure out how to fill it with data through their gql I gave up. And I found another graph db wirtten in Go - elias. Looking at the docs I don't see any mention of spatial data, so I take it there is none at this time?

Also, last commit is 2017 so I am guessing you have moved on to other things?

Lastly, any chance of using Badger as storage backend?

I let the data-miner demo run for some time (2 hours roughly) and get ever-decreasing answer times (4-5 seconds now) when reloading the frontend. As this is just a simple query with a last:50 clause I wonder how the database will perform when using it for something "real"?

Is there any visualization tool that I can use to try and look at the graph?

I wonder what the status is on OSX support?

As an OSX user, I found that the eliasdb console won't work because of the missing implementation of parts of getch in the termutil package of the common module. So I quickly cloned that repository, used a replace statement in the go.mod file, and implemented a very minimal and unfinished Darwin version to play around a bit (not really worth a PR so far).

Next, in my journey, I found that the datamining-example copies over the eliasdb executable into the docker image. I also changed the build script to use a cross compiled version (this could be a small PR).

All in all, I could run all examples on my Mac (strafing in asteroids is cool), and I am thinking about the possibilities for using EliasDB for some projects.

But I read somewhere that ElisaDB was created from scratch using "no dependencies" but the gorilla/websocket. This is true if one does not count for the pretty large dependency on all of those packages below https://devt.de/krotik/common.

While those packages are really cool, they form a substantial part of EliasDB, and this is "just a private gogs server" where all of this is located. This will render the whole EliasDB useless for anybody using it if that repository vanishes. Wouldn't it be better to include the used parts as internal packages or add the whole "common" repo to GitHub?

Is eliasdb production reading? What are the future plans for this project? will it be maintained?

Hi I'm using a p2p network that has an immutable log for each user (secure scuttlebutt). It's a kappa architecture / event sourced p2p database. What I want to do is make it CQRS by not using the database for querying, but instead run through the log inserting messages into eliasdb. That way eliasdb can be the read side only via graphql or EQL.

I'm guessing scaling should be no problem because even though ssb comprises a global network each user generally only stores their data, their friends, and friends of friends. Blobs are stored separately to the log which is json so the log isn't actually that big. I have a huge log stored locally after setting hops=6 to get a lot of data! If it works on my log then it will be fine! It's 1.3 gig, but a normal log would only be 200-300 meg of json messages.

contact messages are used to declare follow, unfollow, or block. I threw them all with directed edges into a graphology graph the other day and there are about 390,000 edges between users. I forget how many nodes. So that's what a large graph would look like. What do you think? Would eliasdb handle that scale, and would it handle the type of queries I would need for a social network?

Noticed that while using a NodeKeyIteration, If nodes were deleted the iterator misses some results.

Is this expected?

An example:

package main

import (
	"fmt"

	"devt.de/krotik/eliasdb/graph"
	"devt.de/krotik/eliasdb/graph/data"
	"devt.de/krotik/eliasdb/graph/graphstorage"
)

func main() {
	TestNodeDeletion()
}

func TestNodeDeletion() {
	gs := graphstorage.NewMemoryGraphStorage("memdb")
	gm := graph.NewGraphManager(gs)

	nodeA1 := data.NewGraphNode()
	nodeA1.SetAttr(data.NodeKey, "nodeA1")
	nodeA1.SetAttr(data.NodeKind, "typeA")

	nodeA2 := data.NewGraphNode()
	nodeA2.SetAttr(data.NodeKey, "nodeA2")
	nodeA2.SetAttr(data.NodeKind, "typeA")

	nodeA3 := data.NewGraphNode()
	nodeA3.SetAttr(data.NodeKey, "nodeA3")
	nodeA3.SetAttr(data.NodeKind, "typeA")

	nodeA4 := data.NewGraphNode()
	nodeA4.SetAttr(data.NodeKey, "nodeA4")
	nodeA4.SetAttr(data.NodeKind, "typeA")

	gm.StoreNode("main", nodeA1)
	gm.StoreNode("main", nodeA2)
	gm.StoreNode("main", nodeA3)
	gm.StoreNode("main", nodeA4)

	fmt.Printf("Node Count: %d\n", gm.NodeCount("typeA"))

	it, _ := gm.NodeKeyIterator("main", "typeA")
	if it != nil {
		for it.HasNext() {
			key := it.Next()
			fmt.Printf("-- Node key = %s lastError = %v\n", key, it.LastError)
			if it.LastError != nil {
				break
			}

			gm.RemoveNode("main", key, "typeA")
		}
	}

	fmt.Printf("Node Count: %d\n", gm.NodeCount("typeA"))

}

When performing the following rest query to an eliasdb running in clustered mode with 2 node:

curl -X POST "https://127.0.0.1:9090/db/v1/graph/my_database/n" -H  "accept: application/json" -H  "Content-Type: application/json" -d "[  {     \"key\": \"3\",     \"kind\": \"Upload\",     \"parcel\": \"12345\"  }]"

I get the following panic:

2019/12/09 08:39:47 http: panic serving 127.0.0.1:52704: interface conversion: interface {} is int, not uint64
goroutine 5100 [running]:
net/http.(*conn).serve.func1(0xc00015e6e0)
        /usr/local/go/src/net/http/server.go:1769 +0x139
panic(0x972200, 0xc000704990)
        /usr/local/go/src/runtime/panic.go:522 +0x1b5
devt.de/krotik/eliasdb/cluster.(*DistributedStorageManager).insertOrUpdate(0xc0007044e0, 0x1, 0x0, 0x963260, 0xc000134400, 0x0, 0x0, 0x0)
        /go/code/cluster/distributedstoragemanager.go:195 +0x7c1
devt.de/krotik/eliasdb/cluster.(*DistributedStorageManager).Insert(0xc0007044e0, 0x963260, 0xc000134400, 0xc000348800, 0xc000026740, 0x30)
        /go/code/cluster/distributedstoragemanager.go:135 +0x4d
devt.de/krotik/eliasdb/hash.NewHTree(0xb31fa0, 0xc0007044e0, 0x0, 0x0, 0x0)
        /go/code/hash/htree.go:128 +0x1c6
devt.de/krotik/eliasdb/graph.(*Manager).getHTree(0xc00010afc0, 0xb31fa0, 0xc0007044e0, 0x2, 0xb31fa0, 0xc0007044e0, 0x8)
        /go/code/graph/helpers.go:434 +0x6c
devt.de/krotik/eliasdb/graph.(*Manager).getIndexHTree(0xc00010afc0, 0xc000686260, 0x9, 0xc0002f05f4, 0x6, 0xc000686201, 0x9fd619, 0x4, 0xa00bba, 0x8, ...)
        /go/code/graph/helpers.go:321 +0x1e4
devt.de/krotik/eliasdb/graph.(*Manager).getNodeIndexHTree(...)
        /go/code/graph/helpers.go:283
devt.de/krotik/eliasdb/graph.(*baseTrans).commitNodes(0xc00012f9c0, 0xc000704330, 0xc000704360, 0xc00065ba48, 0x814329)
        /go/code/graph/trans.go:333 +0x2b9
devt.de/krotik/eliasdb/graph.(*baseTrans).Commit(0xc00012f9c0, 0x0, 0x0)
        /go/code/graph/trans.go:272 +0x19d
devt.de/krotik/eliasdb/api/v1.(*graphEndpoint).handleGraphRequest(0xc000136f60, 0xb2a520, 0xc000668460, 0xc000130f00, 0xc000259860, 0x2, 0x2, 0xa83258, 0xa83260)
        /go/code/api/v1/graph.go:384 +0x530
devt.de/krotik/eliasdb/api/v1.(*graphEndpoint).HandlePOST(0xc000136f60, 0xb2a520, 0xc000668460, 0xc000130f00, 0xc000259860, 0x2, 0x2)
        /go/code/api/v1/graph.go:278 +0x89
devt.de/krotik/eliasdb/api.RegisterRestEndpoints.func1.1(0xb2a520, 0xc000668460, 0xc000130f00)
        /go/code/api/rest.go:159 +0x2b2
net/http.HandlerFunc.ServeHTTP(0xc000102620, 0xb2a520, 0xc000668460, 0xc000130f00)
        /usr/local/go/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xed7820, 0xb2a520, 0xc000668460, 0xc000130f00)
        /usr/local/go/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc0001cea90, 0xb2a520, 0xc000668460, 0xc000130f00)
        /usr/local/go/src/net/http/server.go:2774 +0xa8
net/http.(*conn).serve(0xc00015e6e0, 0xb2b8e0, 0xc00012f740)
        /usr/local/go/src/net/http/server.go:1878 +0x851
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:2884 +0x2f4

Looking at the code, I'd say that either sendDataRequest should return an uint64 always, or, insertOrUpdate should expect to receive non-int64 integers. If possible, the first solution would proably be better, since the empty interface in Go can lead to reduced performance.

Greetings, I saw you can store in either memory or local, however is there a way to encrypt the database?

I followed the steps in the README for downloading EliasDB but the first step

go get -d -v devt.de/common devt.de/eliasdb

returns an error:

can't load package: package devt.de/common: no Go files in /Users/christoph/go/src/devt.de/common

(which appears to be correct because devt.de/common only contains subpackages but not Go files.

I then tried

go get devt.de/common/... devt.de/eliasdb/...

which succeeded and also built and installed eliasdb in $GOPATH/bin.
Are the steps in the README still current? Did I overlook something?

I have a graph where 4 nodes (T1 through T4) are connected like this:

[T1] <-(E2)- [T2] <-(E3)- [T3] -(E4)-> [T4]

The goal of my query is:

Given a T1, find the T4 items.

The kind for T1, T2 and T3 are known but the type for T4 is not. For some reason the follwing query does not work (traversing from the left to right: T1 to T2 to T3 to T4).

get T1 where key = "%s"
    traverse in:E2:out:T2
        traverse in:E3:out:T3
            traverse out:E4:in:
            end
        end
    end

Traversing the nodes and edges in reverse works fine (Right to left).

get T4 where key = "%s"
    traverse in:E4:out:T3
        traverse out:E3:in:T2
            traverse out:E2:in:
            end
        end
    end
show 4:n:key

If I instead start on a node for T2, traverse to T1 and check the contstraint, followed by traversing T2 to T3 to T4 works fine.

get T2
    traverse out:E2:in:T1 where key = "%s"
    end 
    traverse in:E3:out:T3
        traverse out:E4:in:
        end
    end

Alternative, if a use full wildcards for the second and the third traversal, T4 is returned (plus some extra data...)

get T1 where key = "%s"
    traverse in:E2:out:T2
        traverse :::
            traverse :::
            end
        end
    end
show 4:n:key

I can't see why the first query doesn't work. Given that traversing in the reverse works, suggests that the graph is correct. Any idea what the issue is?

It's blockchain data, and there's a key that contains an array, and the keys in that array are almost always different. This blockchain is advancing rapidly, and I can't really say if things will even out too soon (which would enable me to make some kind of struct for this array). So, I noticed the cool trick with the JSON import:

Will this work for any and all JSON?

Say I have 5 gigs of JSON.... what is the best way to feed it to elias?

This is a remarkable piece of work! Just want to say thanks :)

Conversion errors from hex numbers to string with go version 1.16.3.

config/config_test.go:12:32: conversion from untyped int to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)
# devt.de/krotik/eliasdb/graph/graphstorage
graph/graphstorage/diskgraphstorage_test.go:30:32: conversion from untyped int to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)
# devt.de/krotik/eliasdb/graph/util
graph/util/indexmanager.go:42:24: conversion from untyped int to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)
graph/util/indexmanager.go:47:24: conversion from untyped int to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)
graph/util/namesmanager.go:18:20: conversion from untyped int to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)
graph/util/namesmanager.go:23:20: conversion from untyped int to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)
graph/util/namesmanager.go:28:23: conversion from untyped int to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)
graph/util/namesmanager.go:33:21: conversion from untyped int to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)
graph/util/namesmanager.go:38:21: conversion from untyped int to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)
# devt.de/krotik/eliasdb/cluster/manager
cluster/manager/config_test.go:24:32: conversion from untyped int to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)
# devt.de/krotik/eliasdb/storage/paging
storage/paging/pagedstoragefile_test.go:26:32: conversion from untyped int to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)
# devt.de/krotik/eliasdb/storage
storage/diskstoragemanager_test.go:592:32: conversion from untyped int to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)
# devt.de/krotik/eliasdb/storage/file
storage/file/storagefile_test.go:24:32: conversion from untyped int to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)
# devt.de/krotik/eliasdb/server
server/server_test.go:45:32: conversion from untyped int to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)
# devt.de/krotik/eliasdb/graph
graph/globals.go:111:27: conversion from untyped int to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)
graph/globals.go:207:23: conversion from untyped int to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)
graph/globals.go:212:22: conversion from untyped int to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)
graph/globals.go:217:23: conversion from untyped int to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)
graph/globals.go:222:22: conversion from untyped int to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)
graph/graphmanager_test.go:40:32: conversion from untyped int to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)
make: *** [vet] Error 2

The current Web based Term is great.

We could reuse the golang code to make a replacement Web Scripting IDE by doing what Benthos does.

Benthos does the same thing in that its a stream processor with its own interpreter written in golang.

https://github.com/Jeffail/benthos is the golang server.

https://github.com/benthosdev/benthos-lab is the Web based IDE. Its using WASM and reusing the golang code from the server to run.
Demo: https://lab.benthos.dev/

Its just an idea... and so raising it as an option in case someone likes it...

First of all thanks for the nice piece of software. While POC-ing it in our software (https://github.com/nuts-foundation/) I came across a race condition (or rather, go test -race found it) in common's lockfile.go:219 (devt.de/krotik/[email protected]/lockutil/lockfile.go:219) on this line:

lf.running = false

It's triggered by DiskGraphStorage.Close().

You can reproduce it by running the following test with the -race flag:

import (
	"devt.de/krotik/eliasdb/graph"
	"devt.de/krotik/eliasdb/graph/data"
	"devt.de/krotik/eliasdb/graph/graphstorage"
	"github.com/stretchr/testify/assert"
	"os"
	"testing"
)

func TestRacy(t *testing.T) {
	// Cleanup afterwards
	defer os.RemoveAll("racy-testdb")

	gs, err := graphstorage.NewDiskGraphStorage("racy-testdb", false)
	assert.NoError(t, err)
	// Taken from https://github.com/krotik/eliasdb/blob/master/embedding.md
	gm := graph.NewGraphManager(gs)
	node1 := data.NewGraphNode()
	node1.SetAttr("key", "123")
	node1.SetAttr("kind", "mynode")
	node1.SetAttr("name", "Node1")
	node1.SetAttr("text", "The first stored node")
	gm.StoreNode("main", node1)

	// This causes race condition
	assert.NoError(t, gs.Close())
}

The output is as follows:

==================
WARNING: DATA RACE
Write at 0x00c000116568 by goroutine 19:
  devt.de/krotik/common/lockutil.(*LockFile).Finish()
      <redacted>/go/pkg/mod/devt.de/krotik/[email protected]/lockutil/lockfile.go:219 +0x556
  devt.de/krotik/eliasdb/storage.(*ByteDiskStorageManager).Close()
      <redacted>/go/pkg/mod/devt.de/krotik/[email protected]/storage/diskstoragemanager.go:627 +0x4f0
  devt.de/krotik/eliasdb/storage.(*CachedDiskStorageManager).Close()
      <redacted>/go/pkg/mod/devt.de/krotik/[email protected]/storage/cacheddiskstoragemanager.go:229 +0x67
  devt.de/krotik/eliasdb/graph/graphstorage.(*DiskGraphStorage).Close()
      <redacted>/go/pkg/mod/devt.de/krotik/[email protected]/graph/graphstorage/diskgraphstorage.go:202 +0x1e1
  github.com/nuts-foundation/nuts-network/pkg/nextgen/distdoc/racy.TestRacy()
      <redacted>/workspace/nuts-network/pkg/nextgen/distdoc/racy/eliasdb_test.go:28 +0x2e1
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1127 +0x202

Previous read at 0x00c000116568 by goroutine 23:
  devt.de/krotik/common/lockutil.(*LockFile).watch()
      <redacted>/go/pkg/mod/devt.de/krotik/[email protected]/lockutil/lockfile.go:82 +0x124

Goroutine 19 (running) created at:
  testing.(*T).Run()
      /usr/local/go/src/testing/testing.go:1178 +0x796
  testing.runTests.func1()
      /usr/local/go/src/testing/testing.go:1449 +0xa6
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1127 +0x202
  testing.runTests()
      /usr/local/go/src/testing/testing.go:1447 +0x5aa
  testing.(*M).Run()
      /usr/local/go/src/testing/testing.go:1357 +0x4eb
  main.main()
      _testmain.go:43 +0x236

Goroutine 23 (running) created at:
  devt.de/krotik/common/lockutil.(*LockFile).Start()
      <redacted>/go/pkg/mod/devt.de/krotik/[email protected]/lockutil/lockfile.go:185 +0xc5
  devt.de/krotik/eliasdb/storage.initByteDiskStorageManager()
      <redacted>/go/pkg/mod/devt.de/krotik/[email protected]/storage/diskstoragemanager.go:655 +0x1933
  devt.de/krotik/eliasdb/storage.NewByteDiskStorageManager()
      <redacted>/go/pkg/mod/devt.de/krotik/[email protected]/storage/diskstoragemanager.go:235 +0x370
  devt.de/krotik/eliasdb/storage.NewDiskStorageManager()
      <redacted>/go/pkg/mod/devt.de/krotik/[email protected]/storage/diskstoragemanager.go:101 +0x278
  devt.de/krotik/eliasdb/graph/graphstorage.(*DiskGraphStorage).StorageManager()
      <redacted>/go/pkg/mod/devt.de/krotik/[email protected]/graph/graphstorage/diskgraphstorage.go:149 +0x1b4
  devt.de/krotik/eliasdb/graph.(*Manager).getNodeStorageHTree()
      <redacted>/go/pkg/mod/devt.de/krotik/[email protected]/graph/helpers.go:213 +0x421
  devt.de/krotik/eliasdb/graph.(*Manager).storeOrUpdateNode()
      <redacted>/go/pkg/mod/devt.de/krotik/[email protected]/graph/graphmanager_nodes.go:216 +0x1ae
  devt.de/krotik/eliasdb/graph.(*Manager).StoreNode()
      <redacted>/go/pkg/mod/devt.de/krotik/[email protected]/graph/graphmanager_nodes.go:187 +0x2c9
  github.com/nuts-foundation/nuts-network/pkg/nextgen/distdoc/racy.TestRacy()
      <redacted>/workspace/nuts-network/pkg/nextgen/distdoc/racy/eliasdb_test.go:25 +0x285
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1127 +0x202
==================

I wanted to try this db with the memory example but it fails to run because it expects paths to be
"devt.de/..." instead of "github.com/krotik/eliasdb/...".

You should update the project's structure so it can be used as library.

Hi @krotik!

EliasDB looks very promising to become a one-stop GraphQL-DB solution.

However, I think there are some points that could increase the current experience tremendously:

1. Better GraphQL API

For this, you might want to look at how Dgraph handles their GraphQL API.

The API should hide as much backend/query knowledge from the frontend-user as possible. Currently, when using GraphQL, the frontend developer has to have domain knowledge on eliasdb specific syntax like traversal arguments.

Most clients do not care about such power. They want to do the following:

• Filter by fields
• Order by fields
• Paginate

Thats pretty much all IMHO. More complex operations could of course be still available, but maybe optional.

Example GraphQL query:

query {
  queryDatasets(filter: {owner: {in: ["Alice", "Bob"]}}) { // get all datasets whos owner is either alice or bob
    owner
    name
    images(filter: {extension: {eq: "jpg"} }, first: 1) { // retrieve only the first jpg image of these datasets
      filename
      thumbnail
      metadata(order: {asc: key}) { // order metadata alphabetically by their keys
         key
         value
      }
    }
  }
}

The above query is 100% human readable and the client does not need to care about how the underlying DB traverses the graph.

2. Interface/Union Support

Interfaces and Unions are defined on the GraphQL standard. They are important as they reduce query complexity and boilerplate.

Example Query:

query {
  queryFoldersOrFile {
    name
    ... on File {
       extension
    }
    ... on Folder {
        childs {
            __typename
        }
    }
  }
}

3. Bonus Points: Create Database From GraphQL Schema:

Again, this example comes from how dgraph handle things.

Given a GraphQL type schema, eliasdb could bootstrap its whole CRUD-API:

interface UUID {
   id: String! @id //ids are automatically searchable
}

interface Timestamped {
  createdAt: DateTime! @default(add: "now") //default values for different CRUD Actions
  updatedAt: DateTime! @default(add: "now", update: "now")
}

interface Ownable {
  owner: String!
}

type Dataset implements UUID & Timestamped & Ownable {
   name: String! @index // build indices for search
   images: [Image!]! @hasInverse(field: dataset, cascadeDelete: true) // type image has an inverse edge to Dataset, delete all images when a dataset is deleted
}

type Image implements UUID & Timestamped & Ownable {
   filename: String! @index
   extension: String! @index 
   dataset: Dataset! // inverse edge to dataset type required
}

Using the above Info you could completely bootstrap and layout an EliasDB instance and generate a GraphQL CRUD API as described as in 1.

Why not use Dgraph after all? Good Question. Here are the answers:

• Partially Licensed / not Open-Source
• Discontinued / Development has stalled
• Linux Only
• huuuuge Memory requirements
• Missing a lot of features that will not be developed because its discontinued.

But, Dgraph has made a lot of things right and was very close in becoming the GraphQL-DB no-code solution. So we could borrow a lot of concepts to make EliasDB better in that regard.

Happy to hear your thoughts about this!

CJK sentences are not separated by spaces. For now eliasdb can't handle an attempt which intended to search a specific word in some sentence in CJK. It would be great to be able to do that.

When I run the tutorial and enter get line (instead of get Line) I get a list with 1 column of the "Line" keys.

When I then run get Line I don't get any result and the server shows a lot of panics with "Could not take ownership of lockfile db/mainLine.nodes: Could not write lockfile...".

I also wonder what get line would actually mean in that case?

Tested in Linux and Mac.

--- FAIL: TestIndexManagerString (0.00s)
indexmanager_test.go:766: Unexpected string output: IndexManager: 1
1"aaa\b\xf8\xe0&\fdA\x85\x10\xce\xfb+\x06\xee\xe5\xcd" map[testkey:[]]
1"aaabbb" map[testkey:[1]]

you are using a different go docs ? I was hoping to check how many other projects are using eliasdb, but i cant do that with your project.

https://godoc.org/?q=github.com%2Fkrotik%2Feliasdb

$ go mod init example.com/test
go: creating new go.mod: module example.com/test
$ go mod tidy
go: example.com/test imports
        github.com/krotik/eliasdb/eql: github.com/krotik/[email protected]: parsing go.mod:
        module declares its path as: devt.de/krotik/eliasdb
                but was required as: github.com/krotik/eliasdb

The only way that I could fix it was by cloning the repo and added
go mod edit -replace github.com/krotik/eliasdb=../eliasdb

I can see in the source code that there is some support for running eliasdb as a cluster. However, how should this be set up? It seems the CLi has no support for this.

I have a 941MB json file I would like to import into eliasdb, but I get an out of error crash. this is because the importer tries to load the data wholesale in stead of incrementally. A way to do incremental loading from a single json file would be great.

Was there a need to have zero + zero dependencies?

I'm curious about the story behind this DB.. Why it was written and what sets it apart from Cayley and Dgraph?

Hi Krotik

Don't really know if this is supported right now, but I'd rally need to add a condition to the @count function in a conditional statement. For instance, with the dataset provided in the tutorial, let's try to get the stations that are in 2 or more lines whose names begin with D.

I was thinking in something like:

get Station where @count(:Connection::Line where Name beginswith D) > 1

Is there any possibility for this to be added to the EQL language?

Tahnks!

Hello Matthias,
first off, I wanted to compliment you on the project you have created.
To help new users understand the concepts it be nice if you could provide an traveral example for the tutorial_data.
E.g. Show all Stations for Line something

Best regards
Thomas

I haven't seen this in the documentation, but following the tutorial and then inserting nodes with a large-ish string as an attribute intended for Full Text Search (query) - is there a max string length for attribute values? I was trying to to implement a hybrid of graph and full text search, but insertion alone was taking seconds per page (HTML)?

is it possible to build and include arm7 and arm64 binaries with your releases

When benchmarking for a disk storage graph library I got this. The directory is made right before the test. A test graph is made if it does not exist in the storage already. .

panic: Could not take ownership of lockfile unitTestGraph/cagesenseAlarm.nodeidx: Could not write lockfile - read result after writing: 1647273004919593180(expected: 1647273007075243120)

Stacktrace:
goroutine 56 [running]:
devt.de/krotik/eliasdb/storage.initByteDiskStorageManager(0xc0008a9980)
/home/torbjorn/go/pkg/mod/devt.de/krotik/[email protected]/storage/diskstoragemanager.go:657 +0xca8
devt.de/krotik/eliasdb/storage.NewByteDiskStorageManager({0xc00098f5c0, 0x7819e9}, 0x0, 0x0, 0x0, 0x0)
/home/torbjorn/go/pkg/mod/devt.de/krotik/[email protected]/storage/diskstoragemanager.go:235 +0x1ef
devt.de/krotik/eliasdb/storage.NewDiskStorageManager(...)
/home/torbjorn/go/pkg/mod/devt.de/krotik/[email protected]/storage/diskstoragemanager.go:101
devt.de/krotik/eliasdb/graph/graphstorage.(*DiskGraphStorage).StorageManager(0xc000b7a030, {0xc0001b43a8, 0x16}, 0x1)
/home/torbjorn/go/pkg/mod/devt.de/krotik/[email protected]/graph/graphstorage/diskgraphstorage.go:149 +0x125
devt.de/krotik/eliasdb/graph.(*Manager).getIndexHTree(0xc00087c280, {0x77fff7, 0x9}, {0xc00041a158, 0x5}, 0x0, {0x77e7f4, 0x4}, {0x77f747, 0x8})
/home/torbjorn/go/pkg/mod/devt.de/krotik/[email protected]/graph/helpers.go:316 +0x18d
devt.de/krotik/eliasdb/graph.(*Manager).getNodeIndexHTree(...)
/home/torbjorn/go/pkg/mod/devt.de/krotik/[email protected]/graph/helpers.go:283
devt.de/krotik/eliasdb/graph.(*Manager).storeOrUpdateNode(0xc00087c280, {0x77fff7, 0x9}, {0x84d370, 0xc000996590}, 0x2)
/home/torbjorn/go/pkg/mod/devt.de/krotik/[email protected]/graph/graphmanager_nodes.go:243 +0xe9
devt.de/krotik/eliasdb/graph.(*Manager).StoreNode(0xc00087c280, {0x77fff7, 0x9}, {0x84d370, 0xc000996590})
/home/torbjorn/go/pkg/mod/devt.de/krotik/[email protected]/graph/graphmanager_nodes.go:200 +0x17f