Code Monkey home page Code Monkey logo

gold's Introduction

gold

Join the chat at https://gitter.im/linkeddata/gold

gold is a reference Linked Data Platform server for the Solid platform.

Written in Go, based on initial work done by William Waites.

Build Status

Installing

From docker repository:

sudo docker pull linkeddata/gold
sudo docker run -p ip:port:443 linkeddata/gold

Replace ip and port with your host computer's IP address and port number.

To check the status of the container, type:

sudo docker ps

IMPORTANT: if you want to mount a host directory into the container, you can use the -v parameter:

sudo docker run -p ip:port:443 -v /home/user/data:/data linkeddata/gold

This will mount the host directory, /home/user/data, into the container as the /data/ directory. Doing this will allow you to reuse the data directory without worrying about persistence inside the container.

From Github:

  1. Setup Go:

    • Mac OS X: brew install go
    • Ubuntu: sudo apt-get install golang-go
    • Fedora: sudo dnf install golang
  2. Set the GOPATH variable (required by Go):

    mkdir ~/go
    export GOPATH=~/go

    (Optionally consider adding export GOPATH=~/go to your .bashrc or profile).

  3. Check that you have the required Go version (Go 1.4 or later):

    go version
    

    If you don't, please install a more recent version.

  4. Use the go get command to install the server and all the dependencies:

    go get github.com/linkeddata/gold/server
    
  5. Install dependencies:

    • Mac OS X: brew install raptor libmagic
    • Ubuntu: sudo apt-get install libraptor2-dev libmagic-dev
    • Fedora: sudo dnf install raptor2-devel file-devel
  6. (Optional) Install extra dependencies used by the tests:

    go get github.com/stretchr/testify/assert
    

Running the Server

IMPORTANT: Among other things, gold is a web server. Please consider running it as a regular user instead of root. Since gold treats all files equally, and even though uploaded files are not made executable, it will not prevent clients from uploading malicious shell scripts.

Pay attention to the data root parameter, -root. By default, it will serve files from its current directory (so, for example, if you installed it from Github, its data root will be $GOPATH/src/github.com/linkeddata/gold/). Otherwise, make sure to pass it a dedicated data directory to serve, either using a command-line parameter or the config file. Something like: -root=/var/www/data/ or -root=~/data/.

  1. If you installed it from package via go get, you can run it by:
$GOPATH/bin/server -http=":8080" -https=":8443" -debug
  1. When developing locally, you can cd into the repo cloned by go get:
cd $GOPATH/src/github.com/linkeddata/gold

And launch the server by:

go run server/*.go -http=":8080" -https=":8443" -debug -boltPath=/tmp/bolt.db

Alternatively, you can compile and run it from the source dir in one command:

go run $GOPATH/src/github.com/linkeddata/gold/server/*.go -http=":8080" -https=":8443" \
  -root=/home/user/data/ -debug -boltPath=/tmp/bolt.db

Configuration

You can use the provided gold.conf-example file to create your own configuration file, and specify it with the -conf parameter.

cd $GOPATH/src/github.com/linkeddata/gold/
cp gold.conf-example server/gold.conf

# edit the configuration file
nano server/gold.conf

# pass the config file when launching the gold server
$GOPATH/bin/server -conf=$GOPATH/src/github.com/linkeddata/gold/server/gold.conf

To see a list of available options:

~/go/bin/server -help

Some important options and defaults:

  • -conf - Optional path to a config file.

  • -debug - Outputs config parameters and extra logging. Default: false.

  • -root - Specifies the data root directory which gold will be serving. Default: . (so, likely to be $GOPATH/src/github.com/linkeddata/gold/).

  • -http - HTTP port on which the server listens. For local development, the default HTTP port, 80, is likely to be reserved, so pass in an alternative. Default: ":80". Example: -http=":8080".

  • -https - HTTPS port on which the server listens. For local development, the default HTTPS port, 443, is likely to be reserved, so pass in an alternative. Default: ":443". Example: -https=":8443".

Testing

To run the unit tests (assuming you've installed assert via go get github.com/stretchr/testify/assert):

make test

Notes

License

MIT

gold's People

Contributors

csarven avatar dan-f avatar deiu avatar dmitrizagidulin avatar gitter-badger avatar joe-mcnuggets avatar melvincarvalho avatar nicola avatar presbrey avatar reederz avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

gold's Issues

PATCHes which conflict should return "409 Conflict"

I am finding that with gold (databox.me) I am getting a strange error on the client side, "I managed to do the patch you wanted in the server, but the statement you wanted to delete doesn't exist in my local store copy." This I suspects points to the store not properly rejecting with 409 a PATCH whose delete fails. This problem does not happen with ldnode. I am using a linked list, and patching things into the list, only to find the link I was going to replace has gone.

(I don't have an isolated test, but the app I was a using was
https://timbl.databox.me/Public/2015/test/index.html
When hthe client JS console "Uncaught TypeError: Cannot read property 'subject' of undefined" this is because in fact the deleted statement was not in the store. New veresions will have a more explicit exception text)

The problem does not happen with ldnode, though it could be that as ldnode does not support websockets it seems there is less asynchronous activity in the app

ACL UI.

ACL management UI - maybe as a widget/webapp. (does not really depend on gold)

Weird Raptor HTML table produced for FF+tabulator

Accessing the root of a new gold srever using Firefox with tabulator, or

curl -k -H "Accept:text/xml,application/xml,application/xhtml+xml,text/html;q=0.5,text/plain,image/png,/;q=0.1,application/rdf+xml,text/n3,text/turtle

produces a plain HTML table instead of the RDF which should be given, (assuming the resource is primarily a data resource.)

<title>Raptor Graph Serialisation</title>
Subject Predicate Object
https://localhost:9999/ http://www.w3.org/1999/02/22-rdf-syntax-ns#type http://www.w3.org/ns/posix/stat#Directory
https://localhost:9999/ http://www.w3.org/1999/02/22-rdf-syntax-ns#type http://www.w3.org/ns/ldp#BasicContainer
https://localhost:9999/ http://www.w3.org/ns/posix/stat#mtime 1407521961
https://localhost:9999/ http://www.w3.org/ns/posix/stat#size 68

Total number of triples: 4.

API for new accounts (WebIDs).

The server needs a way to issue certs and create profile documents for new users.

An clear issue right now is how/where to serve the API, since the server does not have a restricted URI namespace for application-specific logic such as the API. Basically what URI will the API have, without actually polluting the data URI namespace. A possible solution to the namespace issue could be the use of the ".well-known" namespace.

A starting point would be to "port" the API from https://github.com/linkeddata/ldphp.

Related to #13.

SPARQL parser.

Simple parser for SPARQL. It doesn't need to actually do anything other than parse the query.

Character encoding.

Spaces are not urlencoded by default. Should also check generic utf8 support.

strings more than 245 characters cant be signed

A typical RSA key 2048 bit cannot sign the nonces returned using WebID + RSA

PKCS1v15 uses 11 bytes padding and so can have a max message size of 256 - 11 = 245 bytes

Typically the host + nonce string to sign is of length greater than this.

Would it be possible to use a shorter nonce?

openssl rsautl -sign -in tmp -inkey id_rsa.pem -out sig 
RSA operation error
140315110823584:error:0406C06E:rsa routines:RSA_padding_add_PKCS1_type_1:data too large for key size:rsa_pk1.c:73:

Reading/serializing default turtle.

Do not parse/serialize graph when a request comes for an RDF resource with Accept: text/turtle. We're already storing triples as turtle, so there's no point parsing the file again.

Switch to internal mimeParser/mimeSerializer.

We are currently using libmagic to set the correct CTypes. Unfortunately it does not handle RDF types, so we currently need to work around it [1]. This is especially bad in case of really big files.

To fix it, we should just remove the dependency on libmagic and use the stuff we already have in mime.go together with a the check for "@Prefix" we do in [1](in case a file extension wasn't detected).

[1] https://github.com/linkeddata/gold/blob/master/server.go#L516

Code review: references to example.com

Has this be left there on purpose?

gold/server.go

Lines 328 to 331 in a8eb916

w.Header().Add("Link", brack(resource.Base+"/"+SystemPrefix+"/newCert")+"; rel=\"http://example.org/services#newCert\"")
w.Header().Add("Link", brack(resource.Base+"/"+SystemPrefix+"/accountRecovery")+"; rel=\"http://example.org/services#accountRecovery\"")
w.Header().Add("Link", brack(resource.Base+"/"+SystemPrefix+"/newAccount")+"; rel=\"http://example.org/services#newAccount\"")
w.Header().Add("Link", brack(resource.Base+"/"+SystemPrefix+"/accountInfo")+"; rel=\"http://example.org/services#accountInfo\"")

Crash when adding : <> <> """"1"""" .

This is causing a server crash for me when I run gold in the cloud, latest version

It works locally for me tho

I'll investigate further, but would be interested to know if anyone else can replicate this by uploading that triple to a file.

Make file locking more verbose/semantic.

Tim suggested that when a file is locked, a more complex error response should be sent to the client, possibly indicating the other user who locked the file (maybe using RDF in the body of the response?).

Look at HTTP 423 Locked (WebDAV; RFC 4918)

When using a reverse proxy for SSL termination, gold generates wrong resource URLs

Instead of exposing gold directly to the internet, I want to use NGINX to terminate my SSL connections. The problem is, that when I proxy requests to gold, it thinks that I requested a resource through HTTP and generates wrong resource URLs.

Command to start go server:

go run ~/gopath/src/github.com/linkeddata/gold/server/*.go -http=":8080" -insecure=true -root=/root/jolocom-test-data/ldpc-justas

Nginx host:

server {
        listen 443;
        server_name gold.jolocom.com;

        root html;
        index index.html index.htm;

        ssl on;
        ssl_certificate /etc/nginx/ssl/nginx.crt;
        ssl_certificate_key /etc/nginx/ssl/nginx.key;

        ssl_session_timeout 5m;

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers "HIGH:!aNULL:!MD5 or HIGH:!aNULL:!MD5:!3DES";
        ssl_prefer_server_ciphers on;

        location / {
                proxy_pass         http://localhost:8080/;
                proxy_redirect     off;
                proxy_set_header   Host             $host;
                proxy_set_header   X-Real-IP        $remote_addr;
                proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
        }
}

Querying a resource (notice http instead of https in subjects):

[  4:39PM ]  [ justas@choedankal:~ ]
 $ curl -k -i -H "Accept: application/n-triples" https://gold.jolocom.com/people/justas  
HTTP/1.1 200 OK
Server: nginx/1.4.6 (Ubuntu)
Date: Mon, 27 Apr 2015 14:39:34 GMT
Content-Type: application/n-triples
Transfer-Encoding: chunked
Connection: keep-alive
Accept-Patch: application/json, application/sparql-update
Accept-Post: text/turtle, application/json
Access-Control-Allow-Credentials: true
Access-Control-Expose-Headers: User, Triples, Location, Link, Vary, Last-Modified, Content-Length
Access-Control-Max-Age: 1728000
Allow: OPTIONS, HEAD, GET, PATCH, POST, PUT, MKCOL, DELETE, COPY, MOVE, LOCK, UNLOCK
Etag: "6c15c714783feae55bea5502670c1a91"
Link: <http://gold.jolocom.com/people/justas/,acl>; rel="acl", <http://gold.jolocom.com/people/justas/,meta>; rel="meta"
Link: <http://www.w3.org/ns/ldp#BasicContainer>; rel="type"
Link: <http://www.w3.org/ns/ldp#Resource>; rel="type"
Link: <http://gold.jolocom.com/people/justas/,meta>; rel="meta"
Ms-Author-Via: DAV, SPARQL
Triples: 33
Updates-Via: wss://gold.jolocom.com/
User: 

<http://gold.jolocom.com/people/justas/.acl> <http://www.w3.org/ns/posix/stat#mtime> "1430143653" .
<http://gold.jolocom.com/people/justas/> <http://www.w3.org/ns/ldp#contains> <http://gold.jolocom.com/people/justas/.ttl> .
<http://gold.jolocom.com/people/justas/> <http://www.w3.org/ns/ldp#contains> <http://gold.jolocom.com/people/justas/card> .
<http://gold.jolocom.com/people/justas/> <http://www.w3.org/ns/ldp#contains> <http://gold.jolocom.com/people/justas/card.acl> .
<http://gold.jolocom.com/people/justas/card.ttl> <http://www.w3.org/ns/posix/stat#mtime> "1430143653" .
<http://gold.jolocom.com/people/justas/> <http://www.w3.org/ns/posix/stat#mtime> "1430143653" .
<http://gold.jolocom.com/people/justas/.acl> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/ns/posix/stat#File> .
<http://gold.jolocom.com/people/justas/> <http://www.w3.org/ns/ldp#contains> <http://gold.jolocom.com/people/justas/.acl> .
<http://gold.jolocom.com/people/justas/.acl.ttl> <http://www.w3.org/ns/posix/stat#size> "455" .
<http://gold.jolocom.com/people/justas/card.ttl> <http://www.w3.org/ns/posix/stat#size> "688" .
<http://gold.jolocom.com/people/justas/> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/ns/posix/stat#Directory> .
<http://gold.jolocom.com/people/justas/> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/ns/ldp#BasicContainer> .
<http://gold.jolocom.com/people/justas/.ttl> <http://www.w3.org/ns/posix/stat#size> "332" .
<http://gold.jolocom.com/people/justas/card.acl.ttl> <http://www.w3.org/ns/posix/stat#mtime> "1430143653" .
<http://gold.jolocom.com/people/justas/card.ttl> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/ns/posix/stat#File> .
<http://gold.jolocom.com/people/justas/> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/ns/ldp#Container> .
<http://gold.jolocom.com/people/justas/.acl.ttl> <http://www.w3.org/ns/posix/stat#mtime> "1430143653" .
<http://gold.jolocom.com/people/justas/.ttl> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/ns/posix/stat#File> .
<http://gold.jolocom.com/people/justas/> <http://www.w3.org/ns/ldp#contains> <http://gold.jolocom.com/people/justas/card.ttl> .
<http://gold.jolocom.com/people/justas/> <http://www.w3.org/ns/ldp#contains> <http://gold.jolocom.com/people/justas/.acl.ttl> .
<http://gold.jolocom.com/people/justas/card.acl.ttl> <http://www.w3.org/ns/posix/stat#size> "464" .
<http://gold.jolocom.com/people/justas/card.acl> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/ns/posix/stat#File> .
<http://gold.jolocom.com/people/justas/card.acl> <http://www.w3.org/ns/posix/stat#size> "65" .
<http://gold.jolocom.com/people/justas/.acl> <http://www.w3.org/ns/posix/stat#size> "61" .
<http://gold.jolocom.com/people/justas/.ttl> <http://www.w3.org/ns/posix/stat#mtime> "1430143653" .
<http://gold.jolocom.com/people/justas/card> <http://www.w3.org/ns/posix/stat#size> "61" .
<http://gold.jolocom.com/people/justas/card.acl> <http://www.w3.org/ns/posix/stat#mtime> "1430143653" .
<http://gold.jolocom.com/people/justas/> <http://www.w3.org/ns/posix/stat#size> "4096" .
<http://gold.jolocom.com/people/justas/.acl.ttl> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/ns/posix/stat#File> .
<http://gold.jolocom.com/people/justas/card.acl.ttl> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/ns/posix/stat#File> .
<http://gold.jolocom.com/people/justas/> <http://www.w3.org/ns/ldp#contains> <http://gold.jolocom.com/people/justas/card.acl.ttl> .
<http://gold.jolocom.com/people/justas/card> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/ns/posix/stat#File> .
<http://gold.jolocom.com/people/justas/card> <http://www.w3.org/ns/posix/stat#mtime> "1430143653" .

Link-following SPARQL support.

The server should offer support for LF-SPARQL. We would like to use LF-SPARQL to offload some of the client's work onto the server, and have the server do all the data aggregation from sources all over the Web.

Here's a brief description of how LF-SPARQL is supposed to work. The client (usually a Webapp) posts a query to an LF-SPARQL endpoint, which looks more or less like this:

SELECT ?workspace WHEREVER { <https://deiu.rww.io/profile/card#me> space:storage ?storage; ?storage space:workspace ?workspace . } LIMIT 10

Please note the keyword here is "WHEREVER" instead of "WHERE", which is used to indicate that this is a LF-SPARQL query, and which means that the server should look for data "wherever" it may located -- even on remote servers.

As you might have spotted, the starting point for figuring out where to fetch data from is the WebID: https://deiu.rww.io/profile/card#me.

The algorithm is pretty simple: dereference all resources that are not variables and store the RDF in a common graph (or quad store), then start matching variables within the common graph until all variables are bound.

INFO: Requires Go 1.4+

When following "install from github" instructions on Ubuntu 12 or 14 LTS you get the following errors.

# go get github.com/linkeddata/gold/
# github.com/linkeddata/gold
/root/go/src/github.com/linkeddata/gold/acl.go:52: syntax error: unexpected range, expecting {
/root/go/src/github.com/linkeddata/gold/acl.go:57: syntax error: unexpected range, expecting {
/root/go/src/github.com/linkeddata/gold/acl.go:73: syntax error: unexpected range, expecting {
/root/go/src/github.com/linkeddata/gold/acl.go:80: syntax error: unexpected }
/root/go/src/github.com/linkeddata/gold/acl.go:84: syntax error: unexpected range, expecting {
/root/go/src/github.com/linkeddata/gold/acl.go:100: syntax error: unexpected range, expecting {
/root/go/src/github.com/linkeddata/gold/acl.go:103: non-declaration statement outside function body
/root/go/src/github.com/linkeddata/gold/acl.go:104: syntax error: unexpected range, expecting {
/root/go/src/github.com/linkeddata/gold/acl.go:106: non-declaration statement outside function body
/root/go/src/github.com/linkeddata/gold/acl.go:107: syntax error: unexpected }
/root/go/src/github.com/linkeddata/gold/acl.go:107: too many errors

Ubuntu 14.04 LTS golang-go version is go.1.2.1.

To fix, use gvm, see http://www.hostingadvice.com/how-to/install-golang-on-ubuntu/

Skip step 2 it's redundant, and install dependency apt-get install bison

File browser UI.

The server needs a skin (web app) to display contents of a dir.

Statistics page for tracking e.g number of users

It would be good to have (by default) a public page on gold which gives some anonymous stats like the number of users, mean and median number of triples/user, so we can see ho things grow.

And link it maybe to and from a master list of gold installations (until there a million)

Remove sniffing

Sniffing mime types is a error-prone and nasty way to work, and opens up the system to security attacks. The server should keep track of the mime type of data it has been given.

One way is in the file extension, ensuring the file extension used matches the mime type. Andrei and I discussed this and the result is dumped at
https://www.w3.org/DesignIssues/HTTPFilenameMapping.html
The last method "sweet spot" of filename generation may be the best.

gold should aim to remove sniffing entirely.

Account recovery.

Somehow related to #3 -- users should have a way to recover lost/expired certificates. This could be done through a class password-based login system, or an email recovery system (which could be used in case of losing the password too).

Add LD-Patch generator

Implement code to generate an LD-Patch from two RDF graphs. (that is, it's "diff" for RDF) We also would like this in JavaScript, but maybe GopherJS could be used to generate that from this golang version.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.