Code Monkey home page Code Monkey logo

rdoc's Introduction

rdoc

rdoc (Replicated DOCument): Build better decentralized and offline-first applications in Go

Build Status Package Version

Go Reference

rdoc is a native go implementation of a conflict-free replicated JSON data structure, as introduced by Martin Kleppmann and Alastair R. Beresford in their seminal work [1]. A JSON CRDT is "[...] an algorithm and formal semantics for a JSON data structure that automatically resolves concurrent modifications such that no updates are lost, and such that all replicas converge towards the same state (a conflict-free replicated datatype or CRDT)." [1].

Do you want to learn more about the JSON CRDT data type? This youtube video is a good introduction to the original paper [1] by Martin Kleppmann and Alastair R. Beresford.

Features

  • Simple API; One API call allows the application logic to update and manage coverging JSON replicas in decentralized settings;
  • Supports JSON Patch notation as defined in RFC6902;
  • Supports cbor serialization [WIP; v1.1.0 milestone];

Examples

// starts a new replicated JSON document with an unique ID (in the context of the replicas sample)
doc := Init("doc_replica_1")

// updates the document state with a JSON patch operation:
patch := []byte(`{"op": "add", "path": "/", "value": "user"`)
err := doc.Apply(patch)
if err != nil {
    panic(err)
}

// update the document state with remote operations (i.e. operations executed by a remote replica); 
// remote operations will update the state of the document iif all its dependencies have been applied.  
remotePath := []byte(`[
 {"op": "add", "path": "/", "value": "user", "id":"1.380503024", "deps": [] },
 {"op": "add", "path": "/name", "value": "Jane", "id":"2.1", "deps": ["1.1"] },
 {"op": "add", "path": "/name", "value": "Jane", "id":"2.380503024", "deps": ["1.380503024"] }
]`)

err := doc.Apply(remotePath)
if err != nil {
    panic(err)
}

// Get Doc operations to send over the wire; these operations can be used by
// remote replicas to converge state with `doc1`
doc1Operations := doc.Operations()

// ... apply state from doc1 into doc2, in order for replicas `doc1` and `doc2` to converge

doc2.Apply(doc1Operations)

// ...

// Native Go marshaling/unmarshaling supported 
buffer, err := json.Marshal(*doc)
if err != nil {
    panic(err)
}

References

  1. A Conflict-Free Replicated JSON Datatype (Martin Kleppmann, Alastair R. Beresford)

rdoc's People

Contributors

gpestana avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

rdoc's Issues

JSON patches and arrays

I'm trying to recreate the example shown in Figure 4 from the sited paper: Concurrent editing of an ordered list of characters (i.e., a text document).

I use the following code:

package main

import (
	"encoding/json"
	"fmt"

	jsonpatch "github.com/evanphx/json-patch"
	"github.com/gpestana/rdoc"
)

func main() {
	docA := rdoc.Init("a")
	docB := rdoc.Init("b")

	if err := docA.Apply([]byte(`[
		{"op": "remove", "path": "/x/1", "value": ""},
		{"op": "add", "path": "/x/1", "value": "x"}
	]`)); err != nil {
		panic(err)
	}

	if err := docB.Apply([]byte(`[
		{"op": "add", "path": "/x/0", "value": "y"},
		{"op": "add", "path": "/x/2", "value": "z"}
	]`)); err != nil {
		panic(err)
	}

	opsA, err := docA.Operations()
	if err != nil {
		panic(err)
	}

	opsB, err := docB.Operations()
	if err != nil {
		panic(err)
	}

	docA.Apply(opsB)
	docB.Apply(opsA)

	buildDoc(docA)
	buildDoc(docB)
}

func buildDoc(doc *rdoc.Doc) {
	buffer, err := json.Marshal(*doc)
	if err != nil {
		// handle error
		panic(err)
	}

	patch, err := jsonpatch.DecodePatch(buffer)
	if err != nil {
		panic(err)
	}

	modified, err := patch.Apply([]byte(`{"x": ["a", "b", "c"]}`))
	if err != nil {
		panic(err)
	}
	fmt.Println(string(modified))
}

However, this yields two different documents:

{"x":["y","a","z","x","c"]}
{"x":["y","x","z","b","c"]}

According to the paper, {"x":["y","a","x","z","c"]} would have been correct (which neither gave). More importantly, the documents don't seem to converge.

Upon further inspection, it looks like the operations are the following:

[{"id":"2.6422626","op":"remove","path":"/x/1","deps":[],"value":""},{"id":"3.6422626","op":"add","path":"/x/1","deps":["2.6422626"],"value":"x"},{"id":"2.6488163","op":"add","path":"/x/0","deps":[],"value":"y"},{"id":"3.6488163","op":"add","path":"/x/2","deps":["2.6488163"],"value":"z"}]

[{"id":"2.6488163","op":"add","path":"/x/0","deps":[],"value":"y"},{"id":"3.6488163","op":"add","path":"/x/2","deps":["2.6488163"],"value":"z"},{"id":"2.6422626","op":"remove","path":"/x/1","deps":[],"value":""},{"id":"3.6422626","op":"add","path":"/x/1","deps":["2.6422626"],"value":"x"}]

I suspect the issue is with the premise of using JSON patches for arrays. Given the path references an index in the array, the index would have to be adjusted given the operations that took place before it (similar to who OTs work).

Fix inserting on list

Current implementation is using a list insertAt which uses list indices rather than values. This will create inconsistencies in the final state in cases such as example 4.

How to use patches?

It would be helpful to have an example on how to build a resulting JSON doc. I assume we use the MarshalJSON() method and the jsonpatch library to create the end result. However, this is just a guess.

JSON Merge Patch?

It's an alternative to the more verbose JSONPatch. Do you see any reason why it wouldn't work the same way given your logical clocks? I'm considering implementing it with the same basic structure but creating much smaller patches over the wire.

Examples

Create examples to show how the CRDT interface works.

How are remote patches generated?

I don't see where/how you are extending JSONPatch to include id and deps to be generated on a remote server. Do you have a full example of this?

Example

This looks really interesting. It's great to see crdt being attempted.

It looks like an OT based crdt. I was wondering what is the current status. For example has a test harness been developed yet to prove that all instance types eventually become consistent over time ?

Thanks for this great work. I hope I can help move it along.

CRDT JSON must be immutable

The current implementation is assuming that the document is mutable. This might eventually become troublesome and harder to understand the code due to side effects in the structure.

Think about and implement an API in which the document is immutable. E.g.

doc := crdt.Init()
doc1 := doc.Merge(...)
doc2 := doc1.Apply(...)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.