ueberdosis / hocuspocus Goto Github PK
View Code? Open in Web Editor NEWThe CRDT Yjs WebSocket backend for conflict-free real-time collaboration in your app.
Home Page: https://tiptap.dev/docs/hocuspocus/introduction
License: MIT License
The CRDT Yjs WebSocket backend for conflict-free real-time collaboration in your app.
Home Page: https://tiptap.dev/docs/hocuspocus/introduction
License: MIT License
Description
This is either a server bug, or a documentation issue. Documentation suggests that the requestParameters
payload passed to onConnect
is an object, see:
https://www.hocuspocus.dev/guide/authentication-and-authorization/
However it is actually an instance of URLSearchParams
and attributes must be accessed with get
method, like:
requestParameters.get('access_token') !== 'super-secret-token'
Expected behavior
I think on the server it's more typical for this to be a plain object, leaving the documentation as-is but changing the implementation internally would be the ideal fix.
Description
Packaging bug :
Since package.json states type: module, the require import hint cannot be of a .js file or the server fails to import :
Error [ERR_REQUIRE_ESM]: Must use import to load ES Module: .../node_modules/@hocuspocus/server/dist/hocuspocus-server.js
require() of ES modules is not supported.
require() of .../node_modules/@hocuspocus/server/dist/hocuspocus-server.js from .../hocuspocus-bug.js is an ES module file as it is a .js file whose nearest parent package.json contains "type": "module" which defines all .js files in that package scope as ES modules.
Instead rename hocuspocus-server.js to end in .cjs, change the requiring code to use import(), or remove "type": "module" from .../node_modules/@hocuspocus/server/package.json.
Steps to reproduce the bug
Steps to reproduce the behavior:
Create a single line script :
var h = require('@hocuspocus/server');
Run it
Expected behavior
Import works fine
Environment?
Additional context
Changing @hocuspocus/server/package.json this way
- "require": "./dist/hocuspocus-server.js"
+ "require": "./dist/hocuspocus-server.cjs"
And moving the file appropriately fixes the problem.
The problem I am facing
I'm unable to set a client to read only in methods other than onConnect
The solution I would like
I'd like to have access to the client connection instance in all of the lifecycle hooks.
Alternatives I have considered
I've implemented a workaround that involves passing a getter to the entire Hocuspocus instance to my extension in order to iterate through the connections:
const document = this.getInstance().documents.get(documentName)
document.connections.forEach(({ connection }) => {
const thisClientsVersion = connection.context.SCHEMA_VERSION
if (!thisClientsVersion || thisClientsVersion < requiredVersion) {
this.logger.warn({
label: `${MODULE_NAME}.setOutdatedClientsToReadOnly`,
message: `(${connection.socketId}) Client schema is out of date (v${thisClientsVersion}). Setting it to read only`,
})
connection.readOnly = true
}
})
Additional context
I need to be able to set some connections to read only when certain events happen.
Part of the documentation?
I’ve read the following page of the documentation: https://www.hocuspocus.dev/examples/monaco
Really helpful parts
The demo works perfectly
Hard to understand, missing or misleading
It would be good to have inlined example code for the editor integrations. I'm interested in the Monaco one at the moment. There is no link to the source code and the page source is obfuscated so I couldn't grab it that way.
Hi,
Would it be possible to get the documentName from context and default to parsing the request URL if none is provided? This would help when integrating with express for example
const hookPayload = {
documentName: context?.documentName || Hocuspocus.getDocumentName(request),
requestHeaders: request.headers,
requestParameters: Hocuspocus.getParameters(request),
socketId,
}
Description
onConnect
hooks, no matter what they return or throw, cannot interrupt the hook chain which results in subsequent hooks being run as though everything is OK.
This is especially problematic for auth, as it means there is no way to prevent the createDocument
/createConnection
blocks from being reached:
hocuspocus/packages/server/src/Hocuspocus.ts
Line 176 in 8352196
Steps to reproduce the bug
Steps to reproduce the behavior:
Try to setup the onConnect
example: https://www.hocuspocus.dev/api/on-connect#example
Just throw an error in an extension with onConnect
Observe that it does not prevent downstream onConnect
hooks or the onCreateDocument
hook from running.
Expected behavior
The chain should be interrupted and the connection should be closed (This was previous behavior).
Additional context
I believe this is a regression introduced in 450e5ad
calling .catch
returns a new promise that will continue the chain, just like catching an error. This is described in with examples in the MDN docs: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/catch#using_and_chaining_the_catch_method
Would it be possible to provide a cjs version of the packages?
Hi,
I occurred one more issue during collaboration setup :/ After hocuspocus restarts, existing content from tiptap v2 is sent again, which results doubling the data in the content.
Steps to reproduce:
Error: Front end sends 'Test' content again to hocuspocus, which results with two 'Test' content paragraphs in database / file.
NB: After you restart it again, two 'Test' paragraphs will be sent, which results with 4 'Test' items in the database..
Front end is configured in the following way:
const ydoc = new Y.Doc();
const token = user.token;
const uri = 'ws://localhost';
this.provider = new WebsocketProvider(uri, 'tiptap-example', ydoc, {
params: {
token
}
});
this.provider.on('status', event => {
this.status = event.status;
});
this.editor = new Editor({
extensions: [
...defaultExtensions().filter(extension => extension.config.name !== 'history'),
Underline,
Image,
Document,
Paragraph,
Text,
CollaborationCursor.configure({
provider: this.provider,
user: this.currentUser,
onUpdate: users => {
this.users = users;
},
}),
Collaboration.configure({
provider: this.provider,
document: ydoc
}),
],
});
Thank you for help in advance!
Error:
[2021-04-22T22:09:08.111Z] New connection to "new-doc" …
[2021-04-22T22:09:08.111Z] Created document "new-doc" …
(node:53740) UnhandledPromiseRejectionWarning: Error: "new-doc" is already bound to this RedisPersistence instance
at Object.error.create (~/repo/hocuspocus/node_modules/lib0/error.js:12:28)
at RedisPersistence.bindState (~/repo/hocuspocus/node_modules/y-redis/src/y-redis.js:158:13)
at Redis.onCreateDocument (file://~/repo/hocuspocus/node_modules/@hocuspocus/extension-redis/src/Redis.ts:45:28)
at file://~/repo/hocuspocus/node_modules/@hocuspocus/server/src/Hocuspocus.ts:288:74
...
Setup:
extensions: [
// Log level is not configurable at the moment.
new Logger(),
new Redis({
host: process.env.REDIS_HOST,
port: process.env.REDIS_PORT,
}),
new RocksDB({
// [required] Path to the directory to store the actual data in
path: "./database",
// [optional] Configuration options for theRocksDB adapter, defaults to "{}“
options: {
// This option is only a example. See here for a full list:
// https://www.npmjs.com/package/leveldown#options
createIfMissing: true,
},
}),
],
The problem I am facing
I’m always frustrated when … I need access to my Tiptap extension schema on the server (Hocuspocus or my own API) when the extensions all live in my client package.
The goal is to be able to create documents on the server side with some initial content, which leverages the prosemirrorJSONToYDoc
utility, which requires the schema.
The solution I would like
A method like getSchemaJSON(extensions)
and extensionsFromJSON(jsonSchema)
.
These methods would still need to be run in a browser environment, but having them would make it easy to checkin a copy of the JSON or serve it from an endpoint and then consume it somewhere else.
Alternatives I have considered
We've implemented this by hand and it works, but it feels a little brittle:
import { getSchema } from '@tiptap/core'
export const getSchemaJSON = (extensions) => {
const { spec } = getSchema(extensions)
const marks: any[] = []
const nodes: any[] = []
spec.marks?.forEach((key: string, value: any) => {
marks.push({
name: key,
...value,
})
})
spec.nodes?.forEach((key: string, value: any) => {
nodes.push({
name: key,
...value,
})
})
return {
topNode: spec.topNode,
marks,
nodes,
}
}
import { Node, Mark } from '@tiptap/core'
export const extensionsFromJSON = (jsonSchema) => {
return [
...jsonSchema.marks.map((mark) => Mark.create(mark)),
...jsonSchema.nodes.map((node) => {
const { attrs = {}, ...rest } = node
return Node.create({ ...rest, addAttributes: () => attrs })
}),
]
}
I followed tutorial and done minimal setup, just to make it work, but i am receiving an exception on the front-end.
So, when i return document from onCreateDocument(data) on server side, i am receiving this error on front end:
Exception:
TypeError: Cannot read property 'matchesNode' of null at EditorView.updateStateInner (webpack-internal:///./node_modules/prosemirror-view/dist/index.es.js:4765:43) at EditorView.updateState (webpack-internal:///./node_modules/prosemirror-view/dist/index.es.js:4736:8) at Editor.dispatchTransaction (webpack-internal:///./node_modules/@tiptap/core/dist/tiptap-core.esm.js:2407:19) at EditorView.dispatch (webpack-internal:///./node_modules/prosemirror-view/dist/index.es.js:5004:50) at eval (webpack-internal:///./node_modules/y-prosemirror/src/plugins/sync-plugin.js:301:28) at ProsemirrorBinding.eval [as mux] (webpack-internal:///./node_modules/lib0/mutex.js:37:9) at ProsemirrorBinding._forceRerender (webpack-internal:///./node_modules/y-prosemirror/src/plugins/sync-plugin.js:297:10) at eval (webpack-internal:///./node_modules/y-prosemirror/src/plugins/sync-plugin.js:156:17)
message: "Cannot read property 'matchesNode' of null"
Hocuspocus setup method:
async onCreateDocument(data) {
try {
const prosemirrorDocument = JSON.parse(
`{
"type": "doc",
"content": [{ "type": "paragraph", "content": [] }]
}`
);
// When using the tiptap editor we need the schema to create
// a prosemirror JSON. You can use the `getSchema` method and
// pass it all the tiptap extensions you're using in the frontend
const schema = getSchema([ Document, Paragraph, Text ]);
// Convert the prosemirror JSON to a Y-Doc and simply return it
const result = prosemirrorJSONToYDoc(schema, prosemirrorDocument);
return result;
// return data.document;
} catch(e) {
console.log(e);
}
}
This is front-end setup:
mounted() {
const ydoc = new Y.Doc();
this.provider = new WebsocketProvider(process.env.VUE_APP_WS_ENDPOINT, 'tiptap-collaboration-example', ydoc);
this.provider.on('status', event => {
this.status = event.status;
});
this.editor = new Editor({
extensions: [
Document,
Paragraph,
Text,
CollaborationCursor.configure({
provider: this.provider,
user: this.currentUser,
onUpdate: users => {
this.users = users;
},
}),
Collaboration.configure({ document: ydoc }),
],
});
this.updateCurrentUser({
name: this.getRandomName(),
color: this.getRandomColor()
});
localStorage.setItem('currentUser', JSON.stringify(this.currentUser));
},
Those are the library versions that i am using on the front-end:
"@tiptap/vue-2": "^2.0.0-beta.1",
"@tiptap/starter-kit": "^2.0.0-beta.3",
"@tiptap/core": "^2.0.0-beta.2",
"@tiptap/extension-underline": "^2.0.0-beta.1",
"@tiptap/extension-image": "^2.0.0-beta.1",
"prosemirror-commands": "^1.1.7",
"@tiptap/extension-collaboration": "^2.0.0-beta.3",
"@tiptap/extension-collaboration-cursor": "^2.0.0-beta.3",
"yjs": "^13.5.2",
"y-websocket": "^1.3.11"
Any thoughts what am i doing wrong?
Note to myself, I need to check that:
https://github.com/meatflavourdev/legendary-octo-chainsaw/blob/main/render.yaml
Hi,
I am not sure if this is related to hocuspucus server, or to my config, but when i followed tutorial, i am not able to run my server, because of the added import.
I am receiving the following error:
Error [ERR_REQUIRE_ESM]: Must use import to load ES Module: .../node_modules/@hocuspocus/server/dist/hocuspocus-server.js
require() of ES modules is not supported.
I am using Babel for build..
Update:
I found that this issue is related to the node version:
manuelbieh/geolib#208
After switching to node version 12.12.0, now i got error from your module:
import WebSocket from 'ws';
^^^^^^
SyntaxError: Cannot use import statement outside a module
....
....
I think we’re not using status codes for the close event right now. Leaving this here as a note to check what we’re doing now, and if we can improve that.
There were a few questions already about how to scale hocuspocus. The Redis extension is a start, but we still need to figure a lot of things out. In my opinion, this is how it should be done at the moment:
onConnect
hook or the webhook extension. This node is not connected to the load balancer - Let's call it "manager"This way, all incoming traffic is split equally between the workers. Those sync the changes between themselves and the clients nearly instantaneously. The changes will also be synced over Redis (pub/sub) to the manager node which will handle the storage and any integrations into existing applications. As all of this doesn't need to be instantaneous it's fine if this node is getting slow or runs at max CPU usage. RocksDB and any application integrations could be on two different manager nodes as well.
Things to figure out:
Things to do:
What do you folks think? (cc: @hanspagel)
Any ideas/wishes/opinions on scaling?
Description
I'm trying to contribute to Hocuspocus however I only have access to an M1 machine as my development environment, currently RocksDB will not build on such a machine. It looks like a fix was merged in the last two weeks but we're waiting on a release here:
Once a release is out we can updated RocksDB and all will be well, however it does look like that will mean a major jump from 4.X -> 5.X.
Update: Looks like Gridsome -> Sharp also has an M1 issue blocking install, this could be fixed using resolutions
to force the sharp version
Further update: node-sass @ v5 is also an issue, but it looks like a clean update to v6
Depending on how the promise is rejected in the onConnect hook the server will crash.
If the async function returns Promise.reject();
it's fine, if it returns a rejection message or an Error (should usually be an error) it will crash the server.
Steps to reproduce the bug
const hocuspocus: Hocuspocus = Server.configure({
async onConnect(data) {
return Promise.reject(new Error('paul')); // <== crash
return Promise.reject('paul'); // <== crash
throw new Error('paul'); // <==== crash
return Promise.reject(); // <== works as expected, close the connection and does not crash.
},
});
Hi guys! First off all, thanks for your work. It's amazing!
Description
In my project, I can give users access to edit projects or not.
For example, I gave the user access to edit the project, the user connects to the project and edits it. After a while, I decided to take away the ability to edit from the user. The user should see all updates, but shouldn't edit anything.
How can I immediately inform the client from the server about this, in order to block the editor? I don't need to close his connection, I just need to block his editor.
Also, if the auth token has expired, I would like to close the connection. I can do this with connection.close()
, but in that case, I cannot send an error code and message.
On the server, I check user permission in the onChange hook, and if the user no longer has the ability to edit, I give him a connection.readOnly = true
. And it works, what the user writes is not saved on the server. But I need also block his editor on the client.
And I also check user permission in the onConnect hook, and I need the same instruments here, but currently, I don't have connection as an instance of Connection here, I can only change readOnly mode
This is my code from server
async onConnect (data) {
const permission = await checkPermission(someData);
if (permission.code === 4000) { // read only
data.connection.readOnly = true;
// here I need to send to client message - like 'Read Only'
} else if (permission.code >= 4400) {
// here I need to close connection, but with code and message.
// data.connection.close(permission.code, permission.message) - don't work, because data.connection.close() is not a function
}
}
async onChange (data) {
for (const { connection } of data.document.connections.values()) {
if (connection.socketId === data.socketId) {
const permission = await checkPermission(someData);
if (permission.code === 4300) { // can only read the document
connection.readOnly = true;
// connection.send('test');
} else if (permission.code >= 4400) { // connection must be closed and never reconnect
// here I need to close connection, but with code and message.
// connection.close(permission.code, permission.message);
}
}
}
}
Please, let me know if you have any ideas about it or if you have a better solution for solving this problem.
Thanks!
Description
When a Heading
node is positioned in the middle/end of a document, pressing Enter at the beginning of it's line causes an error.
Steps to reproduce the bug
Steps to reproduce the behavior:
Heading
Expected behavior
The Heading
content is moved down.
Additional context
We noticed this issue with StarterKit v70 as well.
The problem I am facing
I am noticing I sometimes get corrupted documents where certain properties are missing, this is a hard error and results in total loss of work. To combat this I would like to validate my updates being received by clients and optionally do nothing (and tell the sending client to reset their state).
While I would like to fix the underlying issue and not have documents corrupted ever I also would like some resiliency to this as it is somewhat inevitable that bugs like this find their way in.
Given I have a method for validating the JSON representation of my document this is how the flow would work in YJS:
The solution I would like
I can't see any way of fitting this flow into @hocuspocus/server
at the moment as there is no method of listening in to both before and after an update is applied or method of preventing an update being sent to other clients.
Something that does the above would be great but I am open to suggestions on alternatives. The issue I can see with snapshots is I think currently the only methods available are to create a new document out of that snapshot. Which would require replacing the in memory document and telling connected clients to reset their state 🤔
Alternatives I have considered
The above assumes validation on every change, which I don't think should be a problem with a precompiled validation function using something like https://github.com/ajv-validator/ajv. However an alternative flow is to do validation when persisting to long term storage and resetting the server and client state to the last valid one from long term storage when an invalid document is received.
This would look something like:
onChange
Currently I am preventing corrupted documents being saved into long term storage but am lacking a way of force restarting the server and currently connected clients.
Description
According to the docs the way to do authentication is to throw an error in the onConnect
callback if authentication checks fail.
When throwing an error (or rejecting the returned Promise), the connection to the client will be terminated.
Steps to reproduce the bug
Steps to reproduce the behavior:
onConnect
and nothing elsehttps://github.com/ueberdosis/hocuspocus/blob/main/packages/server/src/Hocuspocus.ts#L190
^ This line is never called.
Expected behavior
The connection should be terminated.
It should also not be possible for state to transfer between unauthenticated clients at all, messages received before authentication has completed should probably be held in a queue and emitted once edit: seems like this was the intended behavioronConnect
has resolved.
We want to setup a heartbeat to detect disconnects that is shorter than the 15s period that the awareness protocol uses.
Couldn't create a PR as can't fork the repo but the types
field is incorrect in @hocuspocus/provider
.
This:
hocuspocus/packages/provider/package.json
Line 16 in 9eb6cca
Should be:
"types": "dist/packages/provider/src/index.d.ts",
Hi guys! Thanks for your work!
I have set up a hocusfocus server in my application and it works well. But I have one problem. Socket connection never reconnects.
I opened your sample site and left it open for 2 days, after I came back I can still write something. That's cool. On the network tab, I can see how some socket connections were closed and one is always open.
In my application, I cannot do this. My socket connection always closes after some time of inactivity and never creates a new connection. How can I fix this?
My server code
const server = Server.configure({
async onCreateDocument (data) {
const fieldName = 'default';
if (data.document.isEmpty(fieldName)) {
return;
}
const project = await ProjectService.getById({
id: data.context.projectId,
userId: data.context.userId
});
let prosemirrorJSON = project.document;
if (!prosemirrorJSON) {
const { document } = (new JSDOM(project.body.trim())).window;
const schema = getSchema(extensions);
prosemirrorJSON = DOMParser
.fromSchema(schema)
.parse(document, { preserveWhitespace: true })
.toJSON();
}
return TiptapTransformer.toYdoc(prosemirrorJSON, extensions, fieldName);
},
async onConnect (data) {
const { requestParameters } = data;
const token = requestParameters.get('token');
const user = await UserService.getUserByToken(token);
return {
userId: user.id,
token: token
};
},
async onChange (data) {
if (data.context.ws) {
try {
await CollaborationService.checkPermission({
token: data.context.token,
ws: data.context.ws
});
} catch (error) {
data.context.ws.close(4000, error.Error);
return;
}
}
const save = async () => {
const prosemirrorJSON = TiptapTransformer.fromYdoc(data.document, 'default');
try {
await ProjectService.update({
id: data.context.projectId,
userId: data.context.userId,
document: prosemirrorJSON
});
} catch (error) {
data.context.ws.close(4000, error.Error);
}
};
debounced?.clear();
debounced = debounce(() => save(), 1000);
debounced();
},
timeout: 8640000 // 24hour
});
router.get('/:projectId', async ctx => {
const projectId = parseInt(ctx.params.projectId);
try {
if (ctx.ws) {
const ws = await ctx.ws();
const context = {
projectId: projectId,
ws
};
server.handleConnection(ws, ctx.request, context);
}
} catch (err) {
ctx.bad(400, err);
}
});
Thanks for your help in advance!
We tried to test hocuspocus in a Node.js, but that doesn’t seem to be reliable enough, and it’s hard to write tests like this.
I tried to avoid it, but I think we should set up Cypress to do testing in a browser environment. That’s slower, but it’s probably easier to write tests and see what’s wrong when something fails.
The problem I am facing
I’m used to set HTTP headers for authentication, but that’s not working[1] for WebSocket connections. Currently, we use URL parameters to send tokens, but that doesn’t feel right. Those URLs can land in server logs, and those shouldn’t have tokens.
[1] only through hacks, not in all browsers
The solution I would like
Maybe we could use WebSocket messages to send tokens to the server.
Alternatives I have considered
Cookies, HTTP headers
Additional context
Not my idea, read it here: yjs/y-websocket#8 (comment)
provider.ws.send('ping') but that actually crashed our hocuspocus server due to not being encoded.
Hi There,
I am experimenting with the onConnect hook to see what's possible. I am looking at ways to enforce document naming conventions to ensure they follow a pattern. In our use case it's a course name followed by a lecture date (IE. 'math101.2021apr21')
Anyway I was attempting to do so in the onConnect hook to abort a connection if say if just 'math101' was entered. I had the following thus far as a rudimentary test:
const server = Server.configure({
port:1234,
async onConnect(data) {
const { documentName } = data
const documentSpace = documentName.split('/')
const entityId = documentSpace[1].split('.')
if ( entityId.length !== 2) {
throw new Error('Document Name does not match naming convention')
}
},
I also tried returning a reject()
on the promise.
However it seems when I try the above it does indeed throw the error, but the server process is terminated as well, not just the connection. Any guidance on what I should do differently?
Description
When the onConnect is not synchronous, awareness state leaks to other rooms.
Steps to reproduce the bug
See https://github.com/jperl/yjs-repro and run npm run ok
and npm run bug
.
Expected behavior
I would expect there to only be 1 user state on this document ever.
I have noticed that the provider triggers the synced
and message
events before actually applying the updates.
hocuspocus/packages/provider/src/HocuspocusProvider.ts
Lines 324 to 328 in 5c5f484
For context I am trying to determine when a document has loaded by looking at the contents of it (as I noticed onCreateDocument
fires after the synced
event so I need to listen to synced
and message
and check the contents of my document until it is not empty).
I was seeing a 15 second load time but it was because I was not aware the provider.on()
callbacks trigger before any updates received are actually applied to the document.
For now I have wrapped my code in setTimeout(() => {}, 0)
that checks if the document is loaded but am wondering why it runs in this order / what the use case is for having the event be emitted before the data is in the document?
And if this is intended perhaps this behaviour should be documented as it is somewhat counter intuitive to how the other YJS providers work.
Description
onCreateDocument does not have the initial value from the client
Steps to reproduce the bug
@hocuspocus/server": "^1.0.0-alpha.53
import { Server } from "@hocuspocus/server";
const hocuspocus = Server.configure({
port: 1234,
async onCreateDocument(data) {
console.log("create is empty:", data.document.isEmpty("monaco"));
console.log("create value:", data.document.getText("monaco").toString());
return data.document;
},
async onChange(data) {
console.log("change is empty:", data.document.isEmpty("monaco"));
console.log("change value:", data.document.getText("monaco").toString());
},
});
hocuspocus.listen().then(() => console.log("started"));
Expected behavior
onCreateDocument should have the value
Screenshot, video, or GIF
Description
I tried to install extension-rocksdb on my M1 machine but it fails. Rosetta emulation for terminal doesn't help. It installs fine on intel mac.
Environment?
Additional context
Here is my entire log
Part of the documentation?
On hocuspocus.dev, there are links to /extensions/rocksdb
which need updating to /api/extensions/rocksdb
(i.e. https://www.hocuspocus.dev/api/extensions/rocksdb/ )
No worries, we have you covered! We built an extension that's meant to be used as primary storage:
the [RocksDB extension](/extensions/rocksdb). It's just a couple of lines to integrate.
There are ~4 occurrences: https://github.com/ueberdosis/hocuspocus/search?q=%2Fextensions%2Frocksdb
Additional context
New here. Very excited about hocuspocus and tiptap!
Description
Somewhere in the y-websocket/hocuspocus client/server stack (sorry for not knowing exactly where it comes from) :
When a connection is established with hocuspocus and reset quickly after the instantiation from the client the document's data is not sent properly to the client.
Steps to reproduce the bug see #122 (comment) , more accurate after investigation
hocuspocus + rocksdb server
tiptap + y-websocket frontend
let provider,
ydoc = new Y.Doc(),
docId = 'somedoc';
const registerProvider = () => {
provider = new WebsocketProvider('ws://somehost', docId, ydoc);
};
const resetConnection = () => {
provider.destroy();
registerProvider();
};
// <===== HERE, resetting the provider on the document
// before the document is properly fed from the backend prevents it to ever get its full data
// try with different values between 1 and 100 and you'll reproduce
setTimeout(resetConnection, 10);
new Editor({
extensions: [
StarterKit,
SomeCustomExtension,
Collaboration.configure({ document: ydoc }),
]
});
Expected behavior
Should initialize editor content properly even when connection is resetting a few times quickly
Additional context
I am using firebase authentication to authenticate the user before handing the connection over to hocuspocus, but firebase authentication tokens can change often. So, if the client disconnects and tries to reconnect using y-websocket internal reconnection logic the token may be expired and the server will reject the connection. To avoid that I reset the connection on every onIdTokenChanged , and one of those happen within the first second of a page being loaded.
So I can work around that by preventing the connection reset within the first few seconds, or even better by allowing y-websocket query params to be functions that are called every time a connection attempt is made.
I am not sure if it is bug or a feature... but I think document name should be URL decoded just in case. If user use some unsafe characters in document name, they are auto encoded in URL and they are used as the document name on server.
Some of those characters I tried are: space { } < > | ^. Surprisingly % was ok. You could argue these are very weird chars to have in document name though but I had them and that's how I found out.
hocuspocus/packages/server/src/Hocuspocus.ts
Line 351 in 218143b
Description
Monitor fails to render the entire page when a lot of events are logged (a few thousands in my case, which is actually not that much)
Look at the scrollbar on my screenshot. Only the top of the page is rendered properly, the rest is all blank.
Steps to reproduce the bug
Steps to reproduce the behavior:
Expected behavior
There should be a limit on the number of events to render, optionally with search and pagination. A limit would be fine for a realtime monitor, I'm gessing ~100 events is a sensible limit.
Hi guys!
I am trying to connect your hocuspocus library to my koa server and it does not work as expected.
After some time, the connection to the socket is interrupted and not restored.
I made a connection, as described in the documentation.
I use:
FRONTEND:
BACKEND:
And this is my setup:
const server = Server.configure({
extensions: [
new RocksDB({
path: './tiptap-store'
})
],
timeout: 30000
});
router.get('/:projectId', async ctx => {
try {
if (ctx.ws) {
const ws = await ctx.ws();
server.handleConnection(ws, ctx.request, ctx.params.projectId);
}
} catch (err) {
ctx.bad(400, err);
}
});
Could you please add support for Koa as you have for Express?
Description
I have had to replace @hocuspocus/provider
with y-websocket
as when it shares a document with y-indexeddb
and I replace an existing key in a Y.Map
inside that document the key is deleted (instead of replaced) when using @hocuspocus/provider
.
When using y-websocket
the problem does not exist.
Note:
I do not instantiate the @hocuspocus/provider
until after y-indexeddb
has synced.
Steps to reproduce the bug
I am working on this part, we have a complicated application so working out the best way to reduce this into something that can be verified.
Expected behavior
Should work correctly with y-indexeddb
and replace not delete the key.
Environment?
Additional context
Seeing as it works correctly with y-websocket
instead of @hocuspocs/provider
and they should both be applying / sending updates in the same way, it suggests the issue lies somewhere there?
Note I mentioned corrupted documents in #135, this is not the only source of them (I am getting ones from undo/redo in YJS which are unrelated to hocuspocus) so I am still interested in that topic even if this bug is resolved.
As an additional question, what is the current benefit of using @hocuspocus/provider
over y-websocket
? Is it just something that will be expanded on in future or does it already offer different features?
Hey, I've found a small bug 🐛
If I call my api in onConnect for checking if the user has permission to view/edit the document, only the first connection get the correct state.
Steps to reproduce
const server = Server.configure({
async onConnect(data) {
await new Promise((r) => setTimeout(r, 1337));
}
})
Connect 2 clients from DIFFERENT browsers (otherwise it works fine with Kevin's y-websocket working with Broadcast Channel)
Type in one of the documents
Reload one of the browsers. Now the first changes are gone.
It looks like it's not sending the first message properly with initial state if onConnect is async.
For people using Socket.io it would be great to integrate hocuspocus, like it’s already possible with Express.
When the onConnect hook is really slow and two or more users are connected and one starts to type before the connection is established, everything breaks:
[1] incoming message 0
[1] Caught error while handling a Yjs update Error: Integer out of range!
[1] at Module.readVarUint
hocuspocus/packages/rocksdb/package.json
16: "types": "dist/packages/leveldb/src/index.d.ts"
👆is wrong, it should be 👇
16: "types": "dist/packages/rocksdb/src/index.d.ts"
Sorry couldn't make a PR 😂
That’s not very helpful:
hocuspocus_1 | [2021-05-25T07:09:01.550Z] Created document "App%5CModels%5CNewsletter%3A1" …
hocuspocus_1 | [2021-05-25T07:09:01.621Z] Document "App%5CModels%5CNewsletter%3A1" changed …
hocuspocus_1 | /var/www/node_modules/prosemirror-model/dist/index.js:1368
hocuspocus_1 | if (!json) { throw new RangeError("Invalid input for Node.fromJSON") }
hocuspocus_1 | ^
hocuspocus_1 |
hocuspocus_1 | RangeError: Invalid input for Node.fromJSON
hocuspocus_1 | at Function.fromJSON (/var/www/node_modules/prosemirror-model/dist/index.js:1368:22)
hocuspocus_1 | at prosemirrorJSONToYDoc (/var/www/node_modules/y-prosemirror/dist/y-prosemirror.cjs:1060:27)
hocuspocus_1 | at Prosemirror.toYdoc (file:///var/www/node_modules/@hocuspocus/transformer/dist/hocuspocus-transformer.esm.js:38:20)
hocuspocus_1 | at Tiptap.toYdoc (file:///var/www/node_modules/@hocuspocus/transformer/dist/hocuspocus-transformer.esm.js:10193:39)
hocuspocus_1 | at Webhook.onCreateDocument (file:///var/www/node_modules/@hocuspocus/extension-webhook/dist/hocuspocus-webhook.esm.js:110:68)
hocuspocus_1 | at processTicksAndRejections (node:internal/process/task_queues:96:5)
If the JSON is invalid, the exception should output the JSON. :)
It would be nice if some users could only receive updates and not make changes to the document, like Google Docs read only mode.
Description
Within a custom extension, an onRequest
method with a response and rejected promise crashes Node.js with an unhandled error event. The result is the same for a @hocuspocus/server
onRequest
method when run via Node.js 16.6.1, or a very verbose deprecation warning on 14.17.4.
Steps to reproduce the bug
async onRequest(data: onRequestPayload): Promise<any> {
return new Promise<void>((resolve, reject) => {
const { request, response } = data
response.writeHead(200, { 'Content-Type': 'text/plain' })
response.end('EXTENSION RESPONSE HERE')
return reject()
})
}
cURL
request to the local server, e.g. curl -v http://127.0.0.1:4000/request
[onRequest] Cannot read property 'message' of undefined
events.js:377
throw er; // Unhandled 'error' event
^
Error [ERR_STREAM_WRITE_AFTER_END]: write after end
at writeAfterEnd (_http_outgoing.js:694:15)
at ServerResponse.end (_http_outgoing.js:815:7)
at file:///.../node_modules/@hocuspocus/server/dist/hocuspocus-server.esm.js:1529:26
at processTicksAndRejections (internal/process/task_queues.js:95:5)
Emitted 'error' event on ServerResponse instance at:
at writeAfterEndNT (_http_outgoing.js:753:7)
at processTicksAndRejections (internal/process/task_queues.js:83:21) {
code: 'ERR_STREAM_WRITE_AFTER_END'
}
error Command failed with exit code 1.
Or, for the server method run the example from https://www.hocuspocus.dev/api/on-request and find the same error on the latest Node.js, or the below from version 14:
(node:38547) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'message' of undefined
at file:///.../node_modules/@hocuspocus/server/dist/hocuspocus-server.esm.js:1721:27
at processTicksAndRejections (internal/process/task_queues.js:95:5)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:38547) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 12)
(node:38547) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
Expected behavior
The custom extension onRequest
method sends its response and promise reject to prevent further handling and server output without resulting in a Node.js exit.
Not rejecting the promise prevented crashing, but resulted in a malformed extra "OK" response via
Environment?
I am running into an issue where the redis extension works correctly the first time the server is restarted, however after the initial connection to a document is closed, whenever a new connection is reopened it no longer subscribes to changes of that document.
This is what I think is happening:
y-redis
package, because the y-redis
package is subscribing to the same document that is being edited in memory it is aware of the changesy-redis
package still remembers the document but @hocuspocus/server
forgets the document@hocuspocus/server
creates a new in memory document and the y-redis
package does not subscribe to it because it remembers it.onCreateDocument
hook claims that if you return a document from that hook it is loaded. Which the @hocuspocus/redis
package does (hocuspocus/packages/redis/src/Redis.ts
Line 49 in 5c5f484
hocuspocus/packages/server/src/Hocuspocus.ts
Line 254 in 5c5f484
@hocuspocus/server
which is not the document that y-redis
is subscribed to.I think this can be solved simply by having the @hocuspocus/redis
move the creation of RedisPersistence
to onConnect
and destroy it in onDisconnect
when its the last client:
async onConnect(data) {
if (!this.persistence) {
this.persistence = new yRedis.RedisPersistence(
// @ts-ignore
this.cluster
? { redisClusterOpts: this.configuration }
: { redisOpts: this.configuration });
}
}
async onDisconnect(data) {
if (data.clientsCount === 0) {
this.persistence.destroy();
this.persistence = undefined;
}
}
I have tested this locally and it works for me
The problem I am facing
When a new code release of Hocuspocus goes out, I need to be able to close all open Hocuspocus connections to the old instance and force them to reconnect to the new one. This is necessary to ensure that clients can only connect to 1 Hocuspocus instance at a time for a given doc.
We're using AWS blue/green deployments to cutover traffic, but the websocket connections are not dropped, allowing existing clients to remain connected to the old instance and new clients to the new one.
The solution I would like
An instance method for Hocuspocus to close all open connections. Server.closeAllConnections
This is slightly different from Server.destroy
, which would shutdown the http server and prevent a rollback from being possible.
Alternatives I have considered
We've worked around similar problems in the past using a reference to the instance and reaching into its internals:
this.getInstance().documents.forEach((document) => {
document.connections.forEach((conn: Connection) => {
conn.connection.close()
})
})
Additional context
It's possible that this is solved later down the road once the Hocuspocus scaling work is complete (#87).
Description
When using the onConnect hook to authenticate a client the messages sent by this client before he is disconnected will still be applied to the underlying Y doc.
Steps to reproduce the bug
tiptap collaboration sample should work
const hocuspocus: Hocuspocus = Server.configure({
async onConnect(data) {
// simulate a very slow authentication process that takes 10 seconds (or more if you want to type more)
await new Promise((resolve: Function) => {
setTimeout(() => { resolve(); }, 10000);
});
return Promise.reject();
},
extensions: [
new Logger(),
new RocksDB({
path: './rocksdb',
}),
],
});
return Promise.reject()
Expected behavior
Changes sent by a client that fail to authenticate are dropped into the infinite void of the non-existing-anymore stuff. Or optionally logged somewhere if you don't want to get metaphysical.
Environment?
I use a collaboration extension so that many users can work together on the same document, but users connecting to the same document see different content and they are out of sync with the rest.
In this video you can see how one user opened the same document in 4 browsers - chrome, opera, firefox and safari.
At first they all synced fine, but after restarting all pages the safari connected to another document and was not synced with the other browsers.
Then we switched to another document and saw the same picture, safari connected to another document.
Then we switched to another document, and we can see all the browsers were synchronized.
This also happens when different users connect to the same document.
Please let me know if you need my code.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.