automerge / automerge-repo Goto Github PK
View Code? Open in Web Editor NEWLicense: MIT License
License: MIT License
We should have a demo of how to use automerge repo without libraries
I haven't been able to get the sync server tests to run on windows.
First, there's a line-endings issue with tests.sh
. I can fix this manually by switching my local file from CRLF to LF. There's probably something we can do with the git config so that it doesn't do its normal magic with this one file's line endings on Windows.
Once that is fixed, it complains about imports in the js file.
↯ yarn workspace automerge-repo-sync-server test
yarn workspace v1.22.10
yarn run v1.22.10
$ bash ./scripts/tests.sh
yarn run v1.22.10
yarn run v1.22.10
$ mocha --no-warnings --experimental-specifier-resolution=node --exit
$ node ./src/index.js
/mnt/c/git/pvh/automerge-repo/packages/automerge-repo-sync-server/src/index.js:2
import fs from "fs"
^^
SyntaxError: Unexpected identifier
at Module._compile (internal/modules/cjs/loader.js:723:23)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
at Module.load (internal/modules/cjs/loader.js:653:32)
at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
at Function.Module._load (internal/modules/cjs/loader.js:585:3)
at Function.Module.runMain (internal/modules/cjs/loader.js:831:12)
at startup (internal/bootstrap/node.js:283:19)
at bootstrapNodeJSCore (internal/bootstrap/node.js:623:3)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
Error: Not supported
at Object.exports.doImport (/mnt/c/git/pvh/automerge-repo/node_modules/mocha/lib/nodejs/esm-utils.js:35:41)
at formattedImport (/mnt/c/git/pvh/automerge-repo/node_modules/mocha/lib/nodejs/esm-utils.js:9:28)
at Object.exports.requireOrImport (/mnt/c/git/pvh/automerge-repo/node_modules/mocha/lib/nodejs/esm-utils.js:42:34)
at Object.exports.loadFilesAsync (/mnt/c/git/pvh/automerge-repo/node_modules/mocha/lib/nodejs/esm-utils.js:100:34)
at Mocha.loadFilesAsync (/mnt/c/git/pvh/automerge-repo/node_modules/mocha/lib/mocha.js:447:19)
at singleRun (/mnt/c/git/pvh/automerge-repo/node_modules/mocha/lib/cli/run-helpers.js:125:15)
at exports.runMocha (/mnt/c/git/pvh/automerge-repo/node_modules/mocha/lib/cli/run-helpers.js:190:10)
at Object.exports.handler (/mnt/c/git/pvh/automerge-repo/node_modules/mocha/lib/cli/run.js:370:11)
at innerArgv.then.argv (/mnt/c/git/pvh/automerge-repo/node_modules/yargs/build/index.cjs:443:71)
at process._tickCallback (internal/process/next_tick.js:68:7)
at Function.Module.runMain (internal/modules/cjs/loader.js:834:11)
at startup (internal/bootstrap/node.js:283:19)
at bootstrapNodeJSCore (internal/bootstrap/node.js:623:3)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
./scripts/tests.sh: line 7: kill: (15) - No such process
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed.
Exit code: 1
Command: C:\Program Files\nodejs\node.exe
Arguments: C:\Users\herb\AppData\Roaming\npm\node_modules\yarn\lib\cli.js test
Directory: C:\git\pvh\automerge-repo\packages\automerge-repo-sync-server
Output:
info Visit https://yarnpkg.com/en/docs/cli/workspace for documentation about this command.
In localfirst/relay, the server is written as a typescript class; and there's a separate repo (relay-deployable) that just contains a small js file to instantiate the class and run the server. That setup makes it easy to test the server's functionality without having to spin up separate processes:
import { Server } from './Server'
// ...
describe('Server', () => {
let url: string
let server: Server
beforeAll(async () => {
const port = await getAvailablePort({ port: 3100 })
url = `ws://localhost:${port}`
server = new Server({ port })
await server.listen({ silent: true })
})
afterAll(() => {
server.close()
})
// ...
it('should make a connection', done => {
const { aliceId } = setup()
server.on('introductionConnection', userName => {
expect(userName).toEqual(aliceId)
expect(server.peers).toHaveProperty(aliceId)
expect(server.documentIds).toEqual({})
done()
})
// make a connection
const alice = new WebSocket(`${url}/introduction/${aliceId}`)
})
// ...
})
If that seems like a reasonable approach I'll just refactor the server and tests along those lines, rather than trying to get that script to work cross-platform.
One concern I have about repo is how to manage peers which are running out of sync versions of repo.
For example, an electron app is launched running v1.0. It syncs to a server running the same version.
How can we be sure that both adapters and repo itself can communicate correctly if only one of these is updated?
Talking with @pvh about this, it seems that at a minimum network messages should include some kind of version tag, likely for both repo itself and the adapter employed.
First of all, amazing project! Congrats on the launch of AutoMerge 2.0!
When playing around with this I noticed that TypeScript doesn't pick up on any Repo
properties provided by DocCollection
and EventEmitter
. E.g., Repo.create
turns up as any
.
This can be explained when we look at Repo.d.ts
where extends Mixin(...)
has been replaced with extends Repo_base
, which is of type any
.
When I build from the main branch things seems to be working properly, Repo_base
is not of type any
but has proper typings:
Came here from hypermerge
and really glad to see this project. Pluggability ftw!
I'm getting:
dist/synchronizer/CollectionSynchronizer.js:78
void this.repo.sharePolicy(peerId, documentId).then(okToShare => {
^
TypeError: this.repo.sharePolicy(...).then is not a function
Which is fixed in packages/automerge-repo-sync-server/src/index.js
in the same way as here:
- sharePolicy: (peerId) => false,
+ sharePolicy: async (peerId) => false,
I wanted to start a discussion on how we might go about removing a document/handle from a repo.
It seems to me that there are 2 stages to removing a document:
DocCollection
cache.I would like to propose that we implement stage 1 of the above. The mechanism of locally forgetting a document is useful as a standalone option and would also ease the implementation of stage 2, when we settled on a process.
Happy to put together a forget
PR. If there are any thoughts on naming the API, please let me know.
Peter mentioned the idea of having some sort of RepoId
which would be able to identify a device, and may contain multiple peers.
If this is achievable, a few thoughts on where it might help. The first 3 which spring to mind are:
Whatever the weather, I think being able to group and identify a cluster of Repo
s would be most useful.
At present, the following test fails:
it("cannot find a document which does not exist", async done => {
const { repo } = setup()
const handle = repo.find<TestDoc>("does-not-exist" as DocumentId)
assert.equal(handle.isReady(), false)
handle
.value()
.then(() => {
done(
new Error(
"should not be able to find a document which does not exist"
)
)
})
.catch(() => {
done()
})
})
There likely needs to be some response when a document is not found, but this currently resolves to an empty Automerge doc.
@alexjg I have been running some tests with a sync server, and there were 2 problems which are still there wrt versioning:
I have added some logic to fix the second issue in #142 but it might be better to just keep the error in the adapter for now?
In automerge
the PatchCallback
has been recently updated so that the PatchInfo
parameter includes a source
property.
At the moment, the patch information is returned to the user as below:
Can I propose that we adopt a similar signature to that of the PatchCallback
?
const doc = A.init<T>({
patchCallback: (patches, pathInfo) =>
this.emit("patch", { handle: this, patches, patchInfo }),
})
The demos should not be in /packages
but instead in a top level examples
folder. We should check they are up to date with the current API and ideally build them in CI.
It looks like I won't be able to finish this patch before I go out on holiday, so I'm going to write up my plan and push the branch so that it can either be critiqued or finished (or both).
I'm working on a new format to replace the UUID document IDs used by automerge-repo.
The goal of this change is twofold. First, to introduce a recognizable and consistently parsable URL for Automerge documents that can be stored in an Automerge document. In the future, this URL should support specifiying heads, or perhaps branch IDs, or other kinds of tomfoolery, but for now it's just designed to be a recognizable URL. Second, to allow Automerge-Repo to immediately discard URLs that are either the wrong data type or the result of a transcription error.
The URL format is straightforward:
automerge:<checksummed-bs58-encoded-UUID>
It looks like this:
automerge:3f1w4KRPqEgwCrGGdyM6ATLYfMWo
.
Let's discuss each part of the URL.
First, the scheme, automerge
. I've chosen to use a custom scheme because Automerge is not run over HTTP (though it can tunnel over it via websockets), and because a traditional HTTP url would require us to provide elements like a hostname which doesn't really exist in this context.
Unfortunately, the automerge scheme can't be fetch()'d, at least not as of this writing, and I presume by extension it can't be intercepted by a service worker either. I ran some experiments looking for a scheme that would be accepted by fetch() and so on, and concluded that browser URL parsing inconsistencies and limitations on fetch
meant there was no way to produce an authentic automerge URL that would work reliably without doing things like adding a made-up hostname or calling it HTTPS.
I've kept the UUID as the "deep" representation of the data to maintain some amount of consistency with past versions, and also because using ~128b of entropy to identify some unique resource is pretty much industry standard. I could have thought of a shorter data type (maybe 64 bit is enough) but there's no reason to be cute here.
The encoding serves two purposes. First, a 16 byte UUID encoded into hex is 38 bytes of text. A bs58 encoding is a third shorter, at 24. We then "spend" four of those bytes back to add a checksum which allows us to detect if the URL was copy-pasted with a character missing or (worse) if someone is just passing wildly unrelated values into the system.
I've concluded the URL format should only exist "at the edge" of the library. Internally, on disk, and over the network we should use the most efficient 16 byte binary representation.
This URL format should be stored in the document as a string (for now), but at some point in the future we may add a custom type for it to optimize storage and retrieval of document connections.
Internally to the library, we should use the textual BS58check representation of the UUID for logging and anything user facing. This will allow the user to make visual comparisons to the URL they passed in.
URLs are hard! But this seems like a reasonable plan. The most likely critiques I anticipate are with the bs58check system, and my theory there is that it's better to use something off the shelf than to invent something new.
I welcome your questions / comments.
I had a conversation with @pvh last week about how it would be best to go about managing which documents in a repo are permitted to be shared with which peers. At the moment, there is a sharePolicy
option which takes a peerId
parameter, but this only determines whether or not a peer should be shared with at all.
It seems to me that whether or not a document should be shared with peers depends on factors which may or may not be in Automerge's control or purview. As such, it makes sense to me to take control of that outside of automerge-repo
and allow a user to permit or deny based on their own criteria.
I would love to put together a PR for this, but wanted to discuss a possible API for this before continuing. My preferred route right now would be:
peerId
and a documentId
. This would then be called any time a peer or a document is added to determine whether or not it should be shared.sharePolicy
is set when the peer is added (the callback is evaluated once only), this new callback would be executed each time a new peer joined or a document was added.I am sure there are other methods I have not considered, and would be grateful for peoples' thoughts on the subject.
What follows is a sketch of a proposed approach for supporting authentication & authorization within automerge-repo.
Automerge Repo currently offers no way to authenticate a peer, and very little in the way of access control.
Our current security model is the "Rumplestiltskin rule": If you know a document's ID, you can read that document, and everyone else who knows that ID will accept your changes.
That model is good enough for a surprising number of situations — the ID serves as an unguessable secret "password" for the document — but it has limitations. Without a way to establish a peer's identity, we can't revoke access for an individual peer — say if someone leaves a team, or if a device is lost. And we can't distinguish between read and write permissions, or limit access to specific documents.
An application might implement authentication and authorization in any number of ways, so this should be pluggable — like the existing network and storage adapters.
So initializing a repo might look something like this:
import { SuperCoolAuthProvider } from 'supercool-auth-library'
const authOptions = {
// ...options specific to this type of authentication
}
const auth = new SuperCoolAuthProvider(options)
const repo = new Repo({ network, storage, auth })
An auth provider inherits from this abstract class:
export abstract class AuthProvider extends EventEmitter<AuthProviderEvents> {
/**
* Can this peer prove their identity? The provider implementation will
* use the web socket to communicate with the peer.
*/
abstract async authenticate(peerId: PeerId, socket?: WebSocket): Promise<true | Error> {}
// The following methods may be overriden by the provider and would replace
// the existing `sharePolicy` that we pass to a repo.
/** Should we tell this peer about the existence of this document? */
async okToAdvertise(peerId: PeerId, documentId: DocumentId): Promise<boolean> {
return false
}
/** Should we provide this document (and changes to it) to this peer when asked for it by ID? */
async okToSend(peerId: PeerId, documentId: DocumentId): Promise<boolean> {
return false
}
/** Should we accept changes to this document from this peer? */
async okToReceive(peerId: PeerId, documentId: DocumentId): Promise<boolean> {
return false
}
}
The auth provider's authenticate
method is invoked when the network adapter emits a
peer-candidate
event.
// NetworkSubsystem.ts
networkAdapter.on('peer-candidate', async ({ peerId, channelId, socket }) => {
const { authenticationResult } = await authProvider.authenticate({ peerId, socket })
if (!authenticationResult.isValid) {
const { error } = authenticationResult
this.emit('peer-authentication-failed', { peerId, channelId, error })
} else {
// ...
this.emit('peer', { peerId, channelId })
}
})
TODO
TODO
TODO
I have been looking in to how we might determine if a document is not available on the network. As a side effect, I was investigating how sync messaging is triggered.
At the moment, if we create a document on the repo, we set up a new empty document but don't persist anything to storage. We do, however, immediately start to sync it with other peers (if the sharePolicy allows).
When we find a document which doesn't exist, the same process happens, only the document stays in a requesting state.
Any connected peer will have no way of determining if the first (empty?) sync message is regarding an empty doc which has been created elsewhere and will shortly begin syncing changes or if it is a non existing document that the other peer is looking for.
I would suggest that the most logical solution would be to have a new document only sync over the network if it contains changes?
It might not matter, and we could be happy that a connected peer could respond to the initial sync in either case to say that it the document isn't available from it. If so, we just ignore the message when it arrives at the creating peer.
When we delete a document from a local repo, all it does set the local handle to a deleted state, emit an event and remove from local storage.
While there is a test to ensure that a deleted document can be refetched, I don't think this works as expected. For example, while we remove the local handle from the cache, we don't delete the local synchronizer, nor do we inform the network that we have have deleted the document.
I would suggest that we should remove the local synchronizer and probably inform the network of the deletion. Relevant peers should then remove the peer from their list of peers to synchronize with. This is if particular relevance to a sync server, which would otherwise look to continue syncing changes it receives from elsewhere.
At some point we should add a hook which a peer can use to determine if there is anything else which should be done with a delete message. For example, if you deleted in one tab, you might want to delete in a service worker and all other tabs but just stop syncing from your sync server.
If you try to open a valid documentId that wasn't saved anywhere (and you're not connected to anyone) it should fail gracefully instead of hanging on isReady().
When a handle is changed, the change
event emits as expected. We can then resolve the handle value to get underlying document.
However, if two changes are made in quick succession, only the final value is resolved:
const TEST_ID = "test-document-id" as DocumentId
const handle = new DocHandle<{foo: string}>(TEST_ID, { isNew: true });
handle.on("change", async ({ handle }) => {
const d = await handle.value();
console.log(d);
});
handle.change((doc) => {
doc.foo = "bar";
});
handle.change((doc) => {
doc.foo = "baz";
});
will log
{ foo: 'baz' } <-- should be 'bar'
{ foo: 'baz' }
A simple solution to this would be to include the document in the event parameter object? Happy to put together a PR if this is acceptable.
I tried to fresh install of automerge-repo in a project with yarn v3, it seems like some peer dependencies are not set in packages/
packages etc.
These monorepo packages are missing the following deps/devdeps/peerdeps from their package.json's:
automerge-repo-network-websocket missing:
automerge-repo missing:
The following are all misssing a "@automerge/automerge" dep/devdep/peerdep:
The sync server is a particular application which uses automerge-repo
. I feel that it doesn't belong in this repository. The examples do use it however so we'll need to make sure it's published to NPM in such a way that the examples can depend on it.
We need to say something.
Either
const myInitialValue = {
tasks: [],
filter: "all",
const guaranteeInitialValue = (doc: any) => {
if (!doc.tasks) doc.tasks = []
if (!doc.filter) doc.filter = "all"
return { ...myInitialValue, ...doc }
}
or
<any>
and returns <T>
Ephemeral messages are currently emitted with a channelId which is not used to scope messages in any way.
My understanding of the current setup is that this means that 2 independent peers connected via (for example) a central server, would receive any ephemeral messages that the other sent, even if they were accessing a completely different set of documents.
If this is the case, would it be useful to be able to scope messages to a specific document, such that it is only sent to peers with whom the document is already being synced?
Are there any other scenarios?
There is an uncommon and unlikely-to-effect-the-real-world bug which occurs with the BroadcastChannel network adapter.
When something like Hot Module Replacement causes a page to be initialised a large number of times within a short window of time, any Repo
s on the page are instantiated with a new peerId. All of them join the BroadcastChannel network and they all start syncing with each other. It doesn't take long for this loop to get very much out of hand.
I think this could likely be mitigate by implementing leave
on the adapter, and then exposing a close
or leave
method on Repo itself to allow the user to manually close network connections.
I have tested the issue the latest pre-1.0 release and it also exists then, so this hasn't been introduced by any of our recent efforts.
When more than 2 clients are connected with broadcastchannel, messages bounce around forever and the clients freeze.
I have been looking into how we might be able to remove local documents from a repo (#48) and started to look at how a deleted document might end up returning to a repo inadvertently.
NB. I know that work is being done on authentication and authorisation which will ultimately supplant the current share policy API, but I think this is worth discussing.
The current default share policy is that when 2 peers connect, all of the documents in each repo are synced automatically. This means that when one peer attempts to find
a document which might have previously existed only on the other repo, it is already in its cache, so it finds it immediately.
A better option might be to have a default whereby documents are only shared when requested. However, I am not sure this is possible in the current implementation.
The sharePolicy
is currently consulted (as far as I can tell) whenever a document is added to a repo, whenever a new peer connects and whenever a sync message is received. There are 4 situations which result from this:
find
a document which he doesn't already have. Should the share policy let him request it from Alice?The issue is that we may only want to do some of these activities, and there isn't currently a way to discern which one we are attempting.
I would suggest that we add an additional parameter to the sharePolicy
which allows the user to determine which stage of the process we are at. For example, we could add a 3rd type
argument which is passed in to the sharePolicy
.
export type ShareType = 'CONNECT' | 'CREATE' | 'FIND' | 'REQUEST';
I am sure there are other APIs which could be explored. What do people think?
Obviously this could easily be transferred to @HerbCaudill's auth work as and when
In NetworkSubsystem
, if we see a broadcast message (currently only ephemeral messages), we rebroadcast it to all our other peers.
As noted in the comments, this is not a great gossip protocol. It relies on the network forming a tree. If there are cycles, this approach will loop messages around forever.
See also: Menci/vite-plugin-wasm#37
In the meantime, Orion has a workaround which just recreates the worker after a couple hundred MS but... it's ugly, and it makes using a SharedWorker very inconvenient. There are a few ways we could reduce the grossness of this without waiting for an upstream fix in Chrome but I think it might boil down to including an async import.
We don't 100% have to fix this before 1.0 but it's just so awful I'd like to fit it in if we have time.
At present, if a network adapter fires a peer-disconnected
event, the DocSynchronizer
removes the peer from its list, but maintains it's state.
When a new peer-candidate
event is fired, the DocSychronizer
checks to see if it has a doc state for that peer before adding the peer back on its list. As the state is still present, the peer never gets added.
Happy to issue a pull request for a fix, but I have a question about implementation. Would it be preferred to:
It seems to me that it would be better to preserve the known state so that we would not be losing information already known about the peer's doc. I see that beginSync
method already performs an encode/decode cycle.
NB. I spotted this as I was trying to find a way to reset the state from a network adapter in the case that the connection dropped or a syncMessage
failed to get through.
While running the counter demo on the latest main
, I found that starting (yarn workspace automerge-repo-demo-counter dev
) and stopping (ctrl-c) the dev server a few times while incrementing the counter in the browser causes the saved document (the one in localforage
) to become corrupted1:
caught (in promise) error inflating document chunk ops: invalid changes: changes out of order
Repo.js:36
This persists across reloads (obviously), and only stops once you clear the IndexedDb (where localforage keeps data). I've attached a log containing the localforage::keyvaluepairs
database (with Uint8Array
s encoded to hex).
Ultimately, this leaves the retrieved document handle in the loading state and causes the following error when you click the button and try to update the state:
Handle.js:170 Uncaught (in promise) Error: Cannot change while loading
at DocHandle.change (DocHandle.js:170:19)
at changeDoc (useDocument.js:21:16)
at onClick (App.tsx:14:9)
at HTMLUnknownElement.callCallback2 (react-dom.development.js:4164:14)
at Object.invokeGuardedCallbackDev (react-dom.development.js:4213:16)
at invokeGuardedCallback (react-dom.development.js:4277:31)
at invokeGuardedCallbackAndCatchFirstError (react-dom.development.js:4291:25)
at executeDispatch (react-dom.development.js:9041:3)
at processDispatchQueueItemsInOrder (react-dom.development.js:9073:7)
at processDispatchQueue (react-dom.development.js:9086:5)
Besides fixing the underlying data corruption, it would be nice to surface this error earlier, preferably in Repo.find
, but probably the best you can do is add an error
event to DocHandle
. Otherwise there's no way of knowing that loading the document failed besides listening to the 'error'
event on window
.
It happens pretty consistently, but it's hard to give a specific set of actions. Maybe I'll record a screencast to make it easier for you to reproduce. ↩
I think there is a case for a public api to access a DocHandle
's document without the need for the handle to be "ready".
Whether or not the Handle is ready, it always has a document. It might not be in the expected state, but it exists nonetheless. At any point following handle.value()
resolving, the document is in a valid state.
Hi there,
Awesome to see the recent fixes! However looking at the dist
folder in the 0.0.55
release here https://www.npmjs.com/package/automerge-repo?activeTab=code I can see that some fixes didn't make it.
import { withTimeout, TimeoutError } from "./helpers/withTimeout";
Cheers!
Jun
Peter's aspiration is for this library to provide composable units that can be used independently, or swapped out for alternative implementations.
I'm on board with that aspiration in principle. Clearly the NetworkAdapter
and StorageAdapter
abstractions make a ton of sense, as does the proposed AuthProvider
.
But I still can't picture what it would mean to use DocCollection
, NetworkSubsystem
, EphemeralData
, or StorageSubsystem
independently of Repo
, or what the motivation would be.
In the meantime, we're paying a steep cognitive price for what feels to me like premature abstraction. The work that automerge-repo is doing is not all that complex, but it's hard to see what's actually load-bearing through the maze of event-driven orchestration between all these classes. All this indirection means that takes work to track down who depends on what, what an event is intended to accomplish, and what's part of the public API vs internal only.
I'd like to do one of two things:
Repo
My preference would be to go ahead and do (2). We can always refactor later to reintroduce some abstraction if and when we have an actual use case for it, and in the meantime we'll just be able to see the shape of the thing more clearly.
The current implementation of the Storage Subsystem writes an extra file for each individual change. These files are identified as $docId.incremental.$number
. This works okay if you have only a small number of incremental changes and only a single writer.
We could do better, I think, in two ways. For a storage engine that supports locking and append operations, we could simply append the change onto the end of the document. Automerge is capable of reading additional changes this way (and indeed, this is how we assemble a document at load time.)
For key value engines that don't allow edits, we could store the incremental edits at $docId/incremental/$hashOfChange
. This would allow for a few nice properties. First, we don't have to worry about race conditions in writing (as much). If two agents both write an incremental change, no big deal! (When we go to clean up these incremental keys we should be careful not to delete all keys: more may have arrived from another process while we were working: just delete keys you can guarantee you've got in your current snapshot. You will also need to take a lock while making a snapshot so you don't have two nodes each trying to write a snapshot with disjoint data.)
The cost of this approach is that it will require the storage engines to be able to enumerate all the keys in a sub-namespace. For a filesystem, this is pretty trivial: put the incremental keys in a subdirectory. Other databases will have similar properties. The one place I'm not sure we know how to support with this approach (but that I think we should support) is an S3 backend.
If anyone wants to tinker with this, please feel free to grab me and ask any clarifying questions. Otherwise I'll get around to it at some point, I'm sure.
I have spent some time considering the current NetworkSubsystem
and NetworkAdapter
implementations and I think it's worth starting a discussion on reviewing and simplifying. I admit this is pretty opinionated, but I am not tied to any of these thoughts with any enormous force.
Most of this came from thinking about 2 steps:
ChannelId
TLDR. I think we should lose/replace it.
The ChannelId
(to the best of my understanding) is used in 3 ways. The first is as some kind of ID when we ask the network system and subsequently each adapter to join a "channel". This is done when the system is set up for the first time. We join a SYNC_CHANNEL
channel, then there is never another channel joined, and the SYNC_CHANNEL
is never referred to again. The ID in this case is redundant.
The second is to identify which document is being synced when sync messages are being passed between peers. I think we would be better to just use DocumentId
s in this case, and update the NetworkAdapter
API to match.
The third is to identify if a message is meant for broadcast, if it is ephemeral. In this case the ChannelId
is prefixed with m/
as a way to differentiate it from other message types. I think a clearer way forward would be to specify the message type explicitly as ephemeral
.
Join
At the moment, this is called when the network subsystem is set up, for any channels which are joined. However, when the subsytem is initialised, there are no channels, so nothing is joined. The only time we join is when the repo is initialised for the first time, at the top level of the constructor. I think we could likely do away with join
on both the subsystem and the adapter level; when an adapter is initialised, it should run whatever is needed to join
automatically. The only reason to hang on to it would be in the case that the leave
method is actually useful.
Leave
As far as I can tell, the leave
method in the NetworkSubsystem
is never called. Do we need to actually stop a network adapter? At the moment the suggestion is that we can either leave all adapters (channels, right now) but not individual ones. I can see a world where you might want to be able to stop syncing to a server, for example, but keep syncing between local tabs in your browser/app.
Adapters
At its simplest, a network adapter needs to be able to tell repo when:
It also needs to be able to send messages to those peers.
All of these events currently exist on NetworkAdapter
, but at no point is open
fired by any of the existing adapters. If it did, I think we could bring all of those event types in house
. For example:
open
event.peer-candidate
message type (discussed further below), announcing itself to the network.PeerId
, announce the connection to the DocSynchronizer
and reply with their own peer-candidate
message.I might be underestimating some of the specificity of the initial handshakes required here, but it strikes me that essentially each adapter needs to set up and configure the network, then pass messages between instances of Repo
on one peer, with another.
Messages
So, to messages. I think we should essentially add message types. Off the top of my head, they would be:
peer-candidate
: Sent by a peer when an adapter opens a network connection, or when a new peer connects.
peer-disconnected
: Sent by a peer when it wants to let other peers know that it no longer wants to be a peer, but the network is still active
sync-message
: exchanging sync messages between peers
ephemeral
: sending broadcast messages to all peers
auth
: authentication messages (to fit in with @HerbCaudill's work)
document-not-found
: explicit response to a sync request on a document which a peer does not have, and cannot be found
There may be more here, but the point is that I think that all of these messages could/should be handled by the subsystem, so that the Adapters can be left to set up the network, and then just communicate.
Would love to hear thoughts and corrections on the above. If there is anything I have misunderstood, please don't hesitate to point out
Optional:
Currently the sole form of documentation is the typescript types. This is not nothing but we need more. To start I think we need:
Once the typedoc comments are complete we can publish them on the automerge.org from https://github.com/automerge/automerge.github.io, we already do this for automerge
so it will hopefully be easy to add.
The source of the "how to" guide for automerge is entirely within https://github.com/automerge/automerge.github.io so maybe we should also write the automerge-repo how to guide there?
It seems like automerge-repo currently cannot be used with Deno natively, nor via Deno's npm compatibility layer. When trying to run it natively, Deno complains about all of the imports of .js
files. When trying to import it via npm, the typechecker seems to get hung up on the type of Repo
(specifically its extension of EventEmitter
)
Property 'on' does not exist on type 'Repo'.deno-ts(2339)
Note that the property is definitely there at run-time, but IDEs fail to find it.
main.ts
with the following contents:import {Repo} from "npm:@automerge/[email protected]";
import {MessageChannelNetworkAdapter} from "npm:@automerge/[email protected]";
const messageChannel = new MessageChannel();
const port = messageChannel.port1;
const repoConfig = {
network: [new MessageChannelNetworkAdapter(port)]
};
const repo = new Repo(repoConfig);
repo.on("document", () => true);
Some additional info which is tangentially related: I ran into this issue when trying to wrap the main classes (DocHandle
, Repo
, etc) with my own classes that have the same interface as the ones being wrapped. This included my classes extending EventEmitter. A problem arose because Deno complained (at run-time) I could not invoke super()
on a superclass twice, the superclass being EventEmitter
. Is it an intended use-case of the eventemitter3 library to extend EventEmitter
?
While writing #69 , I noticed that receiveSyncMessage
waits for the doc handle to be ready before calling update
automerge-repo/packages/automerge-repo/src/synchronizer/DocSynchronizer.ts
Lines 135 to 160 in ee50830
receiveSyncMessage
is invoked (indirectly through CollectionSynchronizer
) in an on("message", ...)
listener in Repo.ts
:
automerge-repo/packages/automerge-repo/src/Repo.ts
Lines 98 to 112 in ee50830
However, if we get two messages before doc handle is ready, I don't think there's any guarantee that the two listeners will resume from the await
in the same order they reached it, which could cause sync messages to be processed out of order. I haven't reproduced this yet, so maybe I'm wrong about the spec and someone can correct me.
TLDR: When syncing a large number of batched changes to another peer, a simultaneous update from another peer causes the document to error with a error inflating document chunk ops: mismatching heads
error when saving then loading the document.
First a little context. I have been exploring how automerge-repo
might work when used with an http based network adapter, rather than a tcp based one (eg. websocket). On the server end I am then storing the document in a postgres db using Automerge.save
, then loading it back on each update using Automerge.load
.
Having got it working, I thought it would be sensible to stress test it by creating a large number of changes to sync. This worked without issue, syncing successfully to an additional connected peer. However, due to the latency in the network and the load/save cycle, this took a short while to complete. My next test was to do the same thing, but simultaneously push a change from the additional peer. This is where I first saw the error.
To check that it wasn't an error with my http adapter implementation or improper transaction isolation with postgres, I mocked a version using (slightly modified) existing automerge-repo
adapters. The result was the same. A reproduction is available here.
As this runs on one host, the only modification to the adapter was to introduce some latency so that the batched changes would take a short while to complete.
I suspect that there is a simpler way to recreate the issue, perhaps by simply trying to sync a change from a document which has not yet completed a round of sync messages to update it from another peer.
@alexjg - My hunch is that this is an upstream syncMessage
issue in automerge
itself, but hopefully there is enough information here to help get to the bottom of it.
Is there a vanilla JS browser example available?
Searching for a way to build a multi user collaborative javascript app. Local first and syncing with other clients...
Listening for other clients changes like todo list, chat or similar apps.
There's a convenient hook for loading an array of DocHandles. It's not perfect -- you still need to do some bookkeeping of your own to work wtih it but it's a step in the right direction when working with arrays of documents in react which insists of always having the same number of hooks on every execution.
Code here. Would anyone else want this?
export interface DocIdMap<T> {
[id: DocumentId]: T
}
export function useHandles<T>(docIds?: DocumentId[]): DocIdMap<T> {
const handlersRef = useRef<DocIdMap<DocHandle<T>>>({})
const repo = useRepo()
const [documents, setDocuments] = useState<DocIdMap<T>>({})
if (!docIds) {
return documents
}
const handlers = handlersRef.current
const prevHandlerIds = Object.keys(handlers) as DocumentId[]
docIds.forEach((id) => {
if (handlers[id]) {
return
}
const handler = (handlers[id] = repo.find<T>(id))
handler.value().then((doc) => {
setDocuments((docs) => ({
...docs,
[id]: doc,
}))
})
// TODO: evt.handle.doc isn't awesome
handler.on("change", (evt) => {
setDocuments((docs) => ({
...docs,
[id]: evt.handle.doc,
}))
})
})
prevHandlerIds.forEach((id) => {
if (handlers[id]) {
return
}
const handler = handlers[id]
handler.off("change")
delete handlers[id]
setDocuments((textDocs) => {
const copy = { ...textDocs }
delete copy[id]
return copy
})
})
return documents
}
At the moment, documents which are not found locally are requested from peers. If it's still not found after a period of time, the handle throws a timeout error.
There are a couple of improvements which could be made:
Currently the DocHandle.doc
attribute throws an exception when trying to access it before the document is ready, where ready means (as far as I can tell) one of:
Repo.create()
Repo.find()
and we found something in storageRepo.find()
, nothing was found in storage but a peer sent us some changes for the documentThe intention here is that the user either call DocHandle.isReady
before accessing DocHandle.doc
. There is also an async method DocHandle.value
which waits for the document to become ready and returns the value. This seems confusing to me because it's not clear from the naming or the types what the difference is between doc
and value()
and the relationship to isReady
is oblique.
I think a nicer API would be for DocHandle.doc
to return null
until the document is ready. For typescript users this will signal the difference between value()
and doc
in a way which hopefully will prompt reading the documentation for doc
, which should include a mention of isReady
and value()
.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.