googlearchive / graphd Goto Github PK

View Code? Open in Web Editor NEW

451.0 35.0 53.0 2.58 MB

The Metaweb graph repository server

License: Apache License 2.0

Python 0.17% C 55.28% C++ 0.82% Awk 0.01% Shell 43.48% Makefile 0.03% Go 0.21%

graph-database

graphd's Introduction

Graphd, the Metaweb Graph Repository

For an introduction to graphd, see A Brief Introduction to Graphd

This is not an official Google product.

Building

Build with bazel, like so:

(¬‿¬) bazel build graphd
..............................................
INFO: Analysed target //graphd:graphd (16 packages loaded).
INFO: Found 1 target...
Target //graphd:graphd up-to-date:
  bazel-bin/graphd/graphd
  INFO: Elapsed time: 23.070s, Critical Path: 0.84s
  INFO: Build completed successfully, 377 total actions

Running graphd

To get started:

(¬‿¬) bazel-bin/graphd/graphd -y -d /tmp/db
graphd> write(name = "HAS_KEY")
ok (d119a8c04000dcb38000000000000000)
graphd> read()
ok ((d119a8c04000dcb38000000000000000 null "HAS_KEY" null null null true true 2018-09-07T19:41:57.0000Z))
graphd>

For more details, see graphd.conf(5) and graphd(8)

graphd's People

Contributors

Stargazers

Watchers

Forkers

praveenmunagapati eruditepanda ourobouros leochencipher jithinraj barakmich kochergan teeler nashev tfmorris shaunstanislauslau happy-ferret pierogmorski jonathantito archyver qiuwei hongjun-su hagen666 mbrukman nitingupta910 rajiv256 monad-one tomzhang haoxuu artist100 lcsilver biyituan jiyulongxu martin-ly yfliao dave1667 dairui2 sorcero nextdreamblue arunlodhi animesh-chouhan forestofrain laszlo-kiss moqingxinai dapeng2018 shashishailaj moorthi07 melanyss syyunn zhengxle languageandintelligence neotim isabella232 kewball athanhat tsturge apollohuang1 ztl8702

graphd's Issues

Completion of error handling

Would you like to add more error handling for return values from functions like the following?

Any plan to opensource the mql query engine as well

Hi, is there any plan to release the mql query engine as well?

Startup takes long time, depending on graph size

I have started a graphd database on an iMac (Late 2012, 8 GB RAM, Fusion Drive, macos 10.14) and started to fill it with my word database. It has approximately 63 million nodes and 250 million edges (ranging from pure links to links with integer values and further link attributes).
Filling the graph with gld has varying speeds, from 7000 primitives per hour up to millions per hour. Sometimes insertion appears to halt, once I dared to stopped gld and restarted the computer.
Restarting graphd with the current state of the graph then took 2:09 h, with a graph of then 121 million primitives. It was not a CPU issue, graphd never used more than 2 % of a CPU. But it read gigabytes of SSD/hard disc during that time.
With this kind of startup performance the graph is not ready to use, esp. considering that I have 170 million more primitives to add to the graph until it contains the same amount of data as my original copy in a standard MariaDB.

Is it clear what takes so long during startup?
Is it possible to better configure the system in order to improve startup and/or insert time?
Am I using graphd wrong or do I expect the impossible?

tests don't run

[flavorito:~/github/graphd/test/unit]
(¬‿¬) bazel test :value
INFO: Analysed target //test/unit:value (0 packages loaded).
INFO: Found 1 test target...
FAIL: //test/unit:value (see /usr/local/google/home/rtp/.cache/bazel/_bazel_rtp/46d938b2ea1222dbadb6d3433c0c4d9a/execroot/__main__/bazel-out/k8-fastbuild/testlogs/test/unit/value/test.log)
Target //test/unit:value up-to-date:
  bazel-bin/test/unit/value
INFO: Elapsed time: 0.350s, Critical Path: 0.09s
INFO: Build completed, 1 test FAILED, 2 total actions
//test/unit:value                                                        FAILED in 0.1s
  /usr/local/google/home/rtp/.cache/bazel/_bazel_rtp/46d938b2ea1222dbadb6d3433c0c4d9a/execroot/__main__/bazel-out/k8-fastbuild/testlogs/test/unit/value/test.log

Executed 1 out of 1 test: 1 fails locally.
[flavorito:~/github/graphd/test/unit]
(¬‿¬) cat /usr/local/google/home/rtp/.cache/bazel/_bazel_rtp/46d938b2ea1222dbadb6d3433c0c4d9a/execroot/__main__/bazel-out/k8-fastbuild/testlogs/test/unit/value/test.log
exec ${PAGER:-/usr/bin/less} "$0" || exit 1
-----------------------------------------------------------------------------
/usr/local/google/home/rtp/.cache/bazel/_bazel_rtp/46d938b2ea1222dbadb6d3433c0c4d9a/bazel-sandbox/8517268787008907098/execroot/__main__/bazel-out/k8-fastbuild/bin/test/unit/value.runfiles/__main__/test/unit/value: can't execute /usr/local/google/home/rtp/.cache/bazel/_bazel_rtp/46d938b2ea1222dbadb6d3433c0c4d9a/bazel-sandbox/8517268787008907098/execroot/__main__/bazel-out/k8-fastbuild/bin/test/unit/value.runfiles/test/unit/value.sh or /usr/local/google/home/rtp/.cache/bazel/_bazel_rtp/46d938b2ea1222dbadb6d3433c0c4d9a/bazel-sandbox/8517268787008907098/execroot/__main__/bazel-out/k8-fastbuild/bin/test/unit/value.runfiles/test/unit/value.shsh or /usr/local/google/home/rtp/.cache/bazel/_bazel_rtp/46d938b2ea1222dbadb6d3433c0c4d9a/bazel-sandbox/8517268787008907098/execroot/__main__/bazel-out/k8-fastbuild/bin/test/unit/value.runfiles/test/unit/value.sh.sh

These used to work, needs some bazel mangling.

Native Go version of libgraphdb

A cgo version is possible and I'll probably start with something simple just for experimentation, but it would be interesting to have a native Go version of a similar interface.

Any more documentation

Hello,

I have just started playing around with graphd and it compiles nicely on my Ubuntu 16.04 system.

Is there any more documentation on interfacing and interactive commands?

Perhaps some examples of adding data multiple types of data and querying the data back?

Also, what a but a REST interface with Curl or something?

I'm just trying to get a feel for the capabilities and how to use graphd.

Any help would be greatly appreciated.
Cheers,
Lonnie

doesn't build on mac

might be b/c I'm running Xcode beta, but either way:

rtp-macbookpro2:graphd rtp$ bazel build graphd
Unable to find any JVMs matching version "1.7+".
No Java runtime present, try --request to install.
INFO: Build options have changed, discarding analysis cache.
INFO: Analysed target //graphd:graphd (9 packages loaded).
INFO: Found 1 target...
ERROR: /Users/rtp/github/graphd/libcl/BUILD:6:1: undeclared inclusion(s) in rule '//libcl:libcl':
this rule is missing dependency declarations for the following files included by 'libcl/cl-set-stacktrace.c':
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/string.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/_types.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/cdefs.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_symbol_aliasing.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_posix_availability.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/machine/_types.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/i386/_types.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_pthread/_pthread_types.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/Availability.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/AvailabilityInternal.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types/_size_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types/_null.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types/_rsize_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/machine/types.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/i386/types.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types/_int8_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types/_int16_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types/_int32_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types/_int64_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types/_u_int8_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types/_u_int16_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types/_u_int32_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types/_u_int64_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types/_intptr_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types/_uintptr_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types/_errno_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types/_ssize_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/strings.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/secure/_strings.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/secure/_common.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/secure/_string.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/stdio.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/_stdio.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types/_va_list.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/stdio.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types/_off_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/secure/_stdio.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/stdlib.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/wait.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types/_pid_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types/_id_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/signal.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/appleapiopts.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/machine/signal.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/i386/signal.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/machine/_mcontext.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/i386/_mcontext.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/mach/machine/_structs.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/mach/i386/_structs.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_pthread/_pthread_attr_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types/_sigaltstack.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types/_ucontext.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types/_sigset_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types/_uid_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/resource.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/stdint.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/_types/_uint8_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/_types/_uint16_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/_types/_uint32_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/_types/_uint64_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/_types/_intmax_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/_types/_uintmax_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types/_timeval.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/machine/endian.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/i386/endian.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_endian.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/libkern/_OSByteOrder.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/libkern/i386/_OSByteOrder.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/alloca.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types/_ct_rune_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types/_rune_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types/_wchar_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types/_dev_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types/_mode_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/time.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types/_clock_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types/_time_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types/_timespec.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/unistd.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/unistd.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types/_posix_vdisable.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types/_seek_set.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types/_gid_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types/_useconds_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/select.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types/_fd_def.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types/_suseconds_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types/_fd_setsize.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types/_fd_set.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types/_fd_clr.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types/_fd_isset.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types/_fd_zero.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types/_fd_copy.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_select.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/sys/_types/_uuid_t.h'
  '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/include/gethostuuid.h'
Target //graphd:graphd failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 1.799s, Critical Path: 0.24s
INFO: 0 processes.
FAILED: Build did NOT complete successfully

Newbie - cannot build

Ubuntu 18.04 and bazel 0.23.2
Got this:
~/projects/graphd/graphd$ bazel build graphd
ERROR: /home/davew/.cache/bazel/_bazel_davew/e086e969cb9399a22ef830356847de91/external/go_sdk/BUILD.bazel:52:1: no such target '@io_bazel_rules_go//go:toolchain': target 'toolchain' not declared in package 'go'; however, a source directory of this name exists. (Perhaps add 'exports_files(["toolchain"])' to go/BUILD, or define a filegroup?) defined by /home/davew/.cache/bazel/_bazel_davew/e086e969cb9399a22ef830356847de91/external/io_bazel_rules_go/go/BUILD.bazel and referenced by '@go_sdk//:go_android_arm'
ERROR: While resolving toolchains for target //graphd:graphd: invalid registered toolchain '@go_sdk//:go_android_arm': Analysis failed
ERROR: Analysis of target '//graphd:graphd' failed; build aborted: Analysis failed
INFO: Elapsed time: 3.304s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (10 packages loaded, 28 targets configured)

----end----
Get a similar issue on Win10...

Thanks, Dave W

Remove unnecessary null pointer checks

Extra null pointer checks are not needed in functions like the following.

Freebase / Metaweb Source

Realise this is only slightly related to the repo. But is there any chance that the original freebase web site / web application source code is floating around somewhere and could potentially be open-sourced?

I have a couple projects in the humanitarian / research space and while evaluating options, freebase bubbled up from my memory. This graphd repo is the only reference I can find other than the old google groups to parts of the original freebase project...

reserved identifier violation

I would like to point out that identifiers like “_ADDB_ACCESS_H” and “__GRAPHD_AST__” do not fit to the expected naming convention of the C language standard.
Would you like to adjust your selection for unique names?

Code quality issues

A general bug for any code-quality issues.

See https://hackmd.io/s/HJL3YGm_7 for details, add your own issues there.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.