Code Monkey home page Code Monkey logo

ydotnet's Introduction

YDotNet

YDotNet is a .NET binding for y-crdt. It provides distributed data types that enable real-time collaboration between devices. The library is a thin wrapper around Yrs, taking advantage of the safety and performance of Rust.

💡 Disclaimer: this project is still early, so it may contain bugs and the API is subject to change. Feel free to open an issue if you'd like to report problems or suggest new features.

Demo

Check the following video:

Demo.mp4

Installation

For every scenario, you must start by installing the core of the library.

To do so, in the project directory (where you .csproj lives), execute:

dotnet add package YDotNet

Then, install the platform-specific package in order to get the binaries.

Package Platform
YDotNet.Native.Win32 Windows
YDotNet.Native.Linux Linux
YDotNet.Native.MacOS macOS

And you may also install the following packages to get extra features.

Package Description
YDotNet.Extensions Extension methods to make some operations easier.
YDotNet.Server Provides the hosting infrastructure for all communication channels, like WebSocket.
YDotNet.Server.WebSockets Use WebSockets as the communication channel between clients.
YDotNet.Server.MongoDB Use MongoDB as a persistence layer.
YDotNet.Server.Redis Use Redis as a persistence layer.

Getting Started

YDotNet provides the same shared data types as Yjs. All objects are shared within a Doc and always get modified within the scope of a Transaction.

// Set up the local document with some sample data.
var localDoc = new Doc();
var localText = localDoc.Text("name");

var localTransaction = localDoc.WriteTransaction();
localText.Insert(localTransaction, 0, "Y-CRDT");
localTransaction.Commit();

// Set up the remote document.
var remoteDoc = new Doc();
var remoteText = remoteDoc.Text("name");

// Get the remote document state vector.
var remoteTransaction = remoteDoc.WriteTransaction();
var remoteState = remoteTransaction.StateVectorV1();

// Calculate the state diff between the local and the remote document.
localTransaction = localDoc.ReadTransaction();
var stateDiff = localTransaction.StateDiffV1(remoteState);
localTransaction.Commit();

// Apply the state diff to synchronize the remote document with the local changes.
var result = remoteTransaction.ApplyV1(stateDiff);

// Read the text from the remote document.
var text = remoteText.String(remoteTransaction);

// At this point, the `text` variable is "Y-CRDT" and this demonstrates how the two
// documents synchronized their state.
//
// This example does it locally but the same could be done over the Internet, for example.

Development Setup

To contribute with this library, you'll need to install the following tools:

Then you should clone the y-crdt repository. With the repository cloned and the tools installed, you'll be able to:

  1. Make changes to the Rust or C# library;
  2. Re-build the Rust and C# binaries;
    • Be aware that you'll need to use crate-type=cdylib on the Cargo.toml file to get a dynamic library that's callable by C#.
  3. Test your changes and repeat.

Then you're ready to go! Feel free to contribute, open issues, and ask questions.

Tests

All tests are located in the YDotNet.Tests.Unit project and should be easily runnable using the command:

dotnet test

ydotnet's People

Contributors

lsviana avatar sebastianstehle avatar vdurante avatar japsasuncion avatar akatakritos avatar

Stargazers

Raman Paulau avatar avatar Vasilev Pyotr avatar Tom Waddell avatar Benedikt Nordhoff avatar Hilman Nasrulloh avatar Jacob avatar avatar Cody Mullins avatar avatar YOSHI avatar Kammersgaard avatar Kevin Jahns avatar Tsln avatar Manuel Eisenschink avatar Michael Thomas avatar Ivan avatar avatar avatar Sam G avatar Kurt Ward avatar

Watchers

Kurt Ward avatar Bartosz Sypytkowski avatar avatar avatar

Forkers

goldsam sebastianstehle vdurante akatakritos codymullins japsasuncion

ydotnet's Issues

Improve nullability

The y-crdt library does not null that often. If something fails, you actually get an error indicating what the problem is. Unfortunately the yffi binding is returning null for errors and therefore swallows the actual reason. Very often it is mentioned in the documentation what the problem is.

We should not make the mistake. Lets say you open a transaction in your business logic. What do you do, when the result is null? You have to log this case or throw an exception or something like that and you have to research what the error might have been. It would be more user friendly to just throw the exception. Then the code blows up and is usually handled by an exception handler and logged, like for all other exceptions.

Sometimes the nullability is just wrong, e.g.

public Doc? Clone()
    {
        return ReferenceAccessor.Access(new Doc(DocChannel.Clone(Handle)));
    }

or

public Text? Text(string name)
    {
        var nameHandle = MemoryWriter.WriteUtf8String(name);
        var result = ReferenceAccessor.Access(new Text(DocChannel.Text(Handle, nameHandle)));

        MemoryWriter.Release(nameHandle);

        return result;
    }

In both cases the result can only be null in really buggy situations.

Please assign me, if you agree. Happy to provide a PR.

Create the Text bindings

Implement the bindings for the Text struct based on the exposed bindings.

  • Create class definition
  • ytext
  • ytext_len
  • ytext_string
  • ytext_insert
  • ytext_format
  • ytext_chunks
  • ytext_insert_embed
  • ytext_remove_range
  • ytext_observe
  • ytext_unobserve
  • ytext_event_target
  • ytext_event_path
  • ytext_event_delta
  • ytext_delta_destroy
  • ytext_delta_destroy
  • yobserve_deep
  • yunobserve_deep

For each bound method, the implementation and unit tests must be written.

Exception in Rust code bringing the entire dotnet server down

Hey, sorry to bother once again, but I was wondering if you have any idea for the issue I am facing.

For some reason, every now and then, I get an "exception" in Rust which basically completely breaks the application. The server goes down and has to reboot.

I am trying to pinpoint what is the root cause, but right now I am having a hard time reproducing the issue.

I was wondering if you have any insights on how to prevent the server from going down. If there is any way to capture the rust error, log it and move on.

Thanks!

Fix TODO comments throughout the solution

This list is not final and will be accumulated as items are solved:

  • Make it easier to connect OutputNative size with pointer size in MapEntryNative
  • Remove mentions about *Native counterpart for event classes that don't need it
  • Fix tests for XmlElement.Observe() that weren't using a root-level XmlElement but should be
  • Fix tests for XmlElement.Unobserve() that weren't using a root-level XmlElement but should be
  • Fix tests for XmlElement.Parent() that weren't returning the correct value
  • Fix tests for XmlElement.String() that weren't testing for child XmlText nodes but should be
  • Remove mentions about read-only transactions on writeable functions (they will be tested later, after exception handling is added)
  • Remove mention to test Transaction.Dispose()
  • Remove comment from Output.Object
  • Remove comment from Map.Length()
  • Remove comment from TextChannel
  • Remove comment for Doc.Guid that will not be fixed because the GUID format isn't compatible with C#
  • Remove comment from InputNative because the size of the data can't be reduced or data will be lost
  • Fix the constructor of Output to support null pointer
    • Update usages of new Output() to use ReferenceAccessor.Access(Output)
  • Implement test of getting null value from Map after fixing the Output constructor
  • Replace NotImplemented with NotSupported exception on the EventBranch constructor since all cases are now covered
  • Implement remaining tests marked as [Ignore] because they were waiting for implementations
  • Implement IEnumerator<T>.Reset() on the following classes:
    • ArrayEnumerator
    • MapEnumerator
    • XmlAttributeEnumerator
    • XmlTreeWalkerEnumerator
    • Remove the documentation mention about these classes only supporting one-time enumeration
  • Wrap the XmlElement with an XmlFragment before returning the value.
  • Wrap the XmlText with an XmlFragment before returning the value.
  • Dispose EventSubscription with Unsubscribe().

Create the Branch bindings

Implement the bindings for the Branch struct based on the exposed bindings.

  • Create class definition
  • yobserve_deep
  • yunobserve_deep
  • ytype_get
  • ytype_kind
  • ybranch_write_transaction
  • ybranch_read_transaction
  • Extras
    • Move existing tests for yobserve_deep and yunobserve_deep to the correct namespace

For each bound method, the implementation and unit tests must be written.

Demo client project problems

Using macOS, when running npm install in the Demo/Demo/Client directory, I got the following error:

Error details 
npm ERR! code ERESOLVE
npm ERR! ERESOLVE could not resolve
npm ERR! 
npm ERR! While resolving: [email protected]
npm ERR! Found: @types/[email protected]
npm ERR! node_modules/@types/react
npm ERR!   dev @types/react@"^18.2.15" from the root project
npm ERR!   @types/react@"*" from @types/[email protected]
npm ERR!   node_modules/@types/react-dom
npm ERR!     dev @types/react-dom@"^18.2.7" from the root project
npm ERR! 
npm ERR! Could not resolve dependency:
npm ERR! peer @types/react@"^17.x" from [email protected]
npm ERR! node_modules/react-monaco-editor
npm ERR!   react-monaco-editor@"^0.42.0" from the root project
npm ERR! 
npm ERR! Conflicting peer dependency: @types/[email protected]
npm ERR! node_modules/@types/react
npm ERR!   peer @types/react@"^17.x" from [email protected]
npm ERR!   node_modules/react-monaco-editor
npm ERR!     react-monaco-editor@"^0.42.0" from the root project
npm ERR! 
npm ERR! Fix the upstream dependency conflict, or retry
npm ERR! this command with --force or --legacy-peer-deps
npm ERR! to accept an incorrect (and potentially broken) dependency resolution.
npm ERR! 
npm ERR! 
npm ERR! For a full report see:
npm ERR! /Users/lsviana/.npm/_logs/2023-11-24T22_58_08_125Z-eresolve-report.txt

npm ERR! A complete log of this run can be found in: /Users/lsviana/.npm/_logs/2023-11-24T22_58_08_125Z-debug-0.log

I had to run the command npm install --force to make it work.

NuGet handle

Hi,

because I have accidentally pushed some packages to nuget without my Squidex prefix, I own packages like YDotNet now.

I think the only option is to also make you the owner, but I need your nuget username for that.

EventSubscription could be released with Dispose

I have seen that there is the open task to call dispose on the EventSubscription. We could just use the Dispose method to unsubscribe, e.g.

public IDisposable ObserveAfterTransaction(Action<AfterTransactionEvent> action);

Advantages:

  1. Subscriptions are native resources anyway.
  2. You get warning from code analyzer to dispose the resource.
  3. The API is closer to .NET, e.g. Rx.Net
  4. The API surface is smaller.

YDotnet.Extensions: Exception converting Map to strongly typed model 

    record TodoItem(string Title, bool IsDone);

    [Fact]
    public void CanParseListOfMapsToObjects()
    {
        var doc = new Doc();
        doc.Array("todos");
        using (var txn = doc.WriteTransaction())
        {
            var map = Input.Map(new Dictionary<string, Input>()
            {
                ["Title"] = Input.String("Make dinner"),
                ["IsDone"] = Input.Boolean(false)
            });
            txn.GetArray("todos").InsertRange(txn, 0, map);
        }

        using (var txn = doc.ReadTransaction())
        {
            var array = txn.GetArray("todos");
            var element = array.Get(txn, 0);
            var parsed = element.To<TodoItem>(txn);

            Assert.Equal(new TodoItem("Make dinner", false), parsed);
        }
    }
System.InvalidOperationException: Cannot write a JSON property within an array or as the first JSON token. Current token type is 'StartArray'.

System.InvalidOperationException
Cannot write a JSON property within an array or as the first JSON token. Current token type is 'StartArray'.
   at System.Text.Json.ThrowHelper.ThrowInvalidOperationException(ExceptionResource resource, Int32 currentDepth, Int32 maxDepth, Byte token, JsonTokenType tokenType)
   at System.Text.Json.Utf8JsonWriter.WriteStringByOptionsPropertyName(ReadOnlySpan`1 propertyName)
   at System.Text.Json.Utf8JsonWriter.WritePropertyName(ReadOnlySpan`1 propertyName)
   at YDotNet.Extensions.YDotNetExtensions.<ToJson>g__WriteProperty|5_4(String key, Output value, Utf8JsonWriter jsonWriter, Transaction transaction)
   at YDotNet.Extensions.YDotNetExtensions.<ToJson>g__WriteMap|5_2(Map map, Utf8JsonWriter jsonWriter, Transaction transaction)
   at YDotNet.Extensions.YDotNetExtensions.<ToJson>g__WriteValue|5_5(Output output, Utf8JsonWriter jsonWriter, Transaction transaction)
   at YDotNet.Extensions.YDotNetExtensions.ToJson(Output output, Stream stream, Transaction transaction)
   at YDotNet.Extensions.YDotNetExtensions.To[T](Output output, Transaction transaction)
   at Ballpark.Tests.DocTests.CanParseListOfMapsToObjects() in /Users/mattburke/projects/ballpark/server/Ballpark.Tests/DocTests.cs:line 107
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)

I think this is due to WriteMap doing a WriteStartArray and WriteEndArray instead of WriteStartObject and WriteEndObject:

ydotnet/YDotNet.Extensions/YDotNetExtensions.cs

Line 94 in 0ed99e5

jsonWriter.WriteStartArray();

Would you welcome a PR to add this test and fix the method?

Is there any way to use SignalR?

I was wondering if there is a way of using SignalR instead of WebSockets.

Has anyone achieved this? Is there any example out there on how to do it?

Thanks!

Should inputs and outputs be tied to the transaction?

Some inputs and outputs allocate memory, which has to be freed again. Afaik this does not happen automatically. Therefore every input needs to be released again after the transaction has been completed.

This causes 2 issues IMHO:

  1. It is not so easy to understand when an input need to be released. Because typically, when you add something to a map or array you don't want the items to be released before the container.
  2. The output is even more complicated because you want to have the CLR value longer, but not the output itself.
  3. It is just easy to forget in general.
  4. Sometimes there is no container. Lets say you get a output.Collection or output.Array where you have to loop over the outputs manually and dispose them.

So in general there are not hat much uses cases ,where an input and output needs to live outside the transaction.

What I recommend is the following:

  1. Maintain a list of all allocated resources within an transaction and then release them when the transaction is disposed.
  2. Try to refactor the output so that it resolves the CRL value in the constructor and releases the actual output pointer immediately. I thin there is no need to keep that in memory and the use cases where you need an output without the underlaying value should be very rate.

How would you persist the data in SQL Server?

Just to get an idea, how should I approach the persisting of the document?

Right now, I am using Redis, which acts like a cache and persists the state in binary format for 5mins or so.

How should I approach persisting the data in a readable/searchable format in SQL Server?

ToJson method for Output?

Hi,

I would like to write some business logic, which observes changes on the background and invokes some business logic. I have tested it here: https://github.com/SebastianStehle/ydotnet/blob/main/Demo/Callback.cs

I think right now the type format is pretty complex for that. I would like to have a simple extension method like:

public static T To<T>(this Output output, Doc doc)
{
}

The idea would be to just loop over everything, write the content to a json buffer and deserialize it with System.Text.Json. Not very performance but better than writing your own serializer. What do you think?

Binary Strategy

Hi,

I have seen that you changed the strategy how you distribute binaries in my PR. I understand the intention that you want to have a single package that just works. But I would prefer not to skip the binaries for all platforms, especially for client applications.

So I would do that:

  • YDotnet.Lib or YDotnet.Core for the C# binding
  • YDotnet.Native.XXX for the native libs
  • YDotnet as a meta package that just references YDotnet.Lib and all native packages.

I also noticed that we should build with an older linux version (ubuntu-20.04). The reason is that y-crdt needs libc and the version is dependent on the build platform.

ubuntu-20.04 uses gclib 2.31 which is the same version as the default .NET docker images for 7.0. So if you we build with ubuntu latest, we would link again gclib 2.33, which would then not run with the official images. libc is backwards compatible, so a newer version should not be a problem.

Create the project's infrastructure

Add the basics to get the project started. The items to be added are:

  • Solution
  • Projects
  • Code analyzers
  • Git configuration

After this issue is done, the bindings implementation will be unblocked to start.

Create the Array bindings

Implement the bindings for the Array struct based on the exposed bindings.

  • Create class definition
  • yarray
  • yarray_insert_range
  • yarray_remove_range
  • yarray_len
  • yarray_get
  • yarray_move
  • yarray_iter
  • yarray_iter_destroy
  • yarray_iter_next
  • yarray_observe
  • yarray_unobserve
  • yarray_event_target
  • yarray_event_path
  • yarray_event_delta

For each bound method, the implementation and unit tests must be written.

Unable to load shared library 'yrs' or one of its dependencies.

Hey, I am trying to run the server on a Mac Apple Silicon, but I am getting the following error:

Checking that folder, I can only see a libyrs.dylib file, but no yrs.dylib file.

Thanks!

fail: Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware[1]
      An unhandled exception has occurred while executing the request.
      System.DllNotFoundException: Unable to load shared library 'yrs' or one of its dependencies. In order to help diagnose loading problems, consider setting the DYLD_PRINT_LIBRARIES environment variable: 
      dlopen(/Users/vitor/dev/temp/Yjs/Yjs/bin/Debug/net7.0/runtimes/osx/native/yrs.dylib, 0x0001): tried: '/Users/vitor/dev/temp/Yjs/Yjs/bin/Debug/net7.0/runtimes/osx/native/yrs.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/vitor/dev/temp/Yjs/Yjs/bin/Debug/net7.0/runtimes/osx/native/yrs.dylib' (no such file), '/Users/vitor/dev/temp/Yjs/Yjs/bin/Debug/net7.0/runtimes/osx/native/yrs.dylib' (no such file)
      dlopen(/usr/local/share/dotnet/shared/Microsoft.NETCore.App/7.0.14/yrs.dylib, 0x0001): tried: '/usr/local/share/dotnet/shared/Microsoft.NETCore.App/7.0.14/yrs.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/usr/local/share/dotnet/shared/Microsoft.NETCore.App/7.0.14/yrs.dylib' (no such file), '/usr/local/share/dotnet/shared/Microsoft.NETCore.App/7.0.14/yrs.dylib' (no such file)
      dlopen(/Users/vitor/dev/temp/Yjs/Yjs/bin/Debug/net7.0/yrs.dylib, 0x0001): tried: '/Users/vitor/dev/temp/Yjs/Yjs/bin/Debug/net7.0/yrs.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/vitor/dev/temp/Yjs/Yjs/bin/Debug/net7.0/yrs.dylib' (no such file), '/Users/vitor/dev/temp/Yjs/Yjs/bin/Debug/net7.0/yrs.dylib' (no such file)
      dlopen(yrs.dylib, 0x0001): tried: 'yrs.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OSyrs.dylib' (no such file), '/usr/lib/yrs.dylib' (no such file, not in dyld cache), 'yrs.dylib' (no such file), '/usr/local/lib/yrs.dylib' (no such file), '/usr/lib/yrs.dylib' (no such file, not in dyld cache)
      dlopen(/Users/vitor/dev/temp/Yjs/Yjs/bin/Debug/net7.0/runtimes/osx/native/libyrs.dylib, 0x0001): tried: '/Users/vitor/dev/temp/Yjs/Yjs/bin/Debug/net7.0/runtimes/osx/native/libyrs.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/vitor/dev/temp/Yjs/Yjs/bin/Debug/net7.0/runtimes/osx/native/libyrs.dylib' (no such file), '/Users/vitor/dev/temp/Yjs/Yjs/bin/Debug/net7.0/runtimes/osx/native/libyrs.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64'))
      dlopen(/usr/local/share/dotnet/shared/Microsoft.NETCore.App/7.0.14/libyrs.dylib, 0x0001): tried: '/usr/local/share/dotnet/shared/Microsoft.NETCore.App/7.0.14/libyrs.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/usr/local/share/dotnet/shared/Microsoft.NETCore.App/7.0.14/libyrs.dylib' (no such file), '/usr/local/share/dotnet/shared/Microsoft.NETCore.App/7.0.14/libyrs.dylib' (no such file)
      dlopen(/Users/vitor/dev/temp/Yjs/Yjs/bin/Debug/net7.0/libyrs.dylib, 0x0001): tried: '/Users/vitor/dev/temp/Yjs/Yjs/bin/Debug/net7.0/libyrs.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/vitor/dev/temp/Yjs/Yjs/bin/Debug/net7.0/libyrs.dylib' (no such file), '/Users/vitor/dev/temp/Yjs/Yjs/bin/Debug/net7.0/libyrs.dylib' (no such file)
      dlopen(libyrs.dylib, 0x0001): tried: 'libyrs.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibyrs.dylib' (no such file), '/usr/lib/libyrs.dylib' (no such file, not in dyld cache), 'libyrs.dylib' (no such file), '/usr/local/lib/libyrs.dylib' (no such file), '/usr/lib/libyrs.dylib' (no such file, not in dyld cache)
      dlopen(/Users/vitor/dev/temp/Yjs/Yjs/bin/Debug/net7.0/runtimes/osx/native/yrs, 0x0001): tried: '/Users/vitor/dev/temp/Yjs/Yjs/bin/Debug/net7.0/runtimes/osx/native/yrs' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/vitor/dev/temp/Yjs/Yjs/bin/Debug/net7.0/runtimes/osx/native/yrs' (no such file), '/Users/vitor/dev/temp/Yjs/Yjs/bin/Debug/net7.0/runtimes/osx/native/yrs' (no such file)
      dlopen(/usr/local/share/dotnet/shared/Microsoft.NETCore.App/7.0.14/yrs, 0x0001): tried: '/usr/local/share/dotnet/shared/Microsoft.NETCore.App/7.0.14/yrs' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/usr/local/share/dotnet/shared/Microsoft.NETCore.App/7.0.14/yrs' (no such file), '/usr/local/share/dotnet/shared/Microsoft.NETCore.App/7.0.14/yrs' (no such file)
      dlopen(/Users/vitor/dev/temp/Yjs/Yjs/bin/Debug/net7.0/yrs, 0x0001): tried: '/Users/vitor/dev/temp/Yjs/Yjs/bin/Debug/net7.0/yrs' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/vitor/dev/temp/Yjs/Yjs/bin/Debug/net7.0/yrs' (no such file), '/Users/vitor/dev/temp/Yjs/Yjs/bin/Debug/net7.0/yrs' (no such file)
      dlopen(yrs, 0x0001): tried: 'yrs' (no such file), '/System/Volumes/Preboot/Cryptexes/OSyrs' (no such file), '/usr/lib/yrs' (no such file, not in dyld cache), 'yrs' (no such file), '/usr/local/lib/yrs' (no such file), '/usr/lib/yrs' (no such file, not in dyld cache)
      dlopen(/Users/vitor/dev/temp/Yjs/Yjs/bin/Debug/net7.0/runtimes/osx/native/libyrs, 0x0001): tried: '/Users/vitor/dev/temp/Yjs/Yjs/bin/Debug/net7.0/runtimes/osx/native/libyrs' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/vitor/dev/temp/Yjs/Yjs/bin/Debug/net7.0/runtimes/osx/native/libyrs' (no such file), '/Users/vitor/dev/temp/Yjs/Yjs/bin/Debug/net7.0/runtimes/osx/native/libyrs' (no such file)
      dlopen(/usr/local/share/dotnet/shared/Microsoft.NETCore.App/7.0.14/libyrs, 0x0001): tried: '/usr/local/share/dotnet/shared/Microsoft.NETCore.App/7.0.14/libyrs' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/usr/local/share/dotnet/shared/Microsoft.NETCore.App/7.0.14/libyrs' (no such file), '/usr/local/share/dotnet/shared/Microsoft.NETCore.App/7.0.14/libyrs' (no such file)
      dlopen(/Users/vitor/dev/temp/Yjs/Yjs/bin/Debug/net7.0/libyrs, 0x0001): tried: '/Users/vitor/dev/temp/Yjs/Yjs/bin/Debug/net7.0/libyrs' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/vitor/dev/temp/Yjs/Yjs/bin/Debug/net7.0/libyrs' (no such file), '/Users/vitor/dev/temp/Yjs/Yjs/bin/Debug/net7.0/libyrs' (no such file)
      dlopen(libyrs, 0x0001): tried: 'libyrs' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibyrs' (no such file), '/usr/lib/libyrs' (no such file, not in dyld cache), 'libyrs' (no such file), '/usr/local/lib/libyrs' (no such file), '/usr/lib/libyrs' (no such file, not in dyld cache)
      
         at YDotNet.Native.Document.DocChannel.NewWithOptions(DocOptionsNative options)
         at YDotNet.Document.Doc.CreateDoc(DocOptions options)
         at YDotNet.Document.Doc..ctor(DocOptions options)
         at YDotNet.Document.Doc..ctor()
         at YDotNet.Server.Internal.DocumentContainer.LoadCoreAsync()
         at YDotNet.Server.Internal.DocumentContainer.LoadInternalAsync(IDocumentCallback documentCallback, IDocumentManager documentManager)
         at YDotNet.Server.Internal.DocumentContainer.ApplyUpdateReturnAsync[T](Func`2 action)
         at YDotNet.Server.DefaultDocumentManager.GetStateVectorAsync(DocumentContext context, CancellationToken ct)
         at YDotNet.Server.WebSockets.YDotNetSocketMiddleware.<HandleSyncAsync>b__11_0(WebSocketEncoder encoder, Boolean context, ClientState state, CancellationToken ct)
         at YDotNet.Server.WebSockets.ClientState.WriteLockedAsync[T](T state, Func`5 action, CancellationToken ct)
         at YDotNet.Server.WebSockets.YDotNetSocketMiddleware.HandleSyncAsync(ClientState state, CancellationToken ct)
         at YDotNet.Server.WebSockets.YDotNetSocketMiddleware.InvokeAsync(HttpContext httpContext, String documentName)
         at YDotNet.Server.WebSockets.YDotNetSocketMiddleware.InvokeAsync(HttpContext httpContext, String documentName)
         at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.InvokeCore(HttpContext context, PathString matchedPath, PathString remainingPath)
         at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddlewareImpl.Invoke(HttpContext context)

Create the Map bindings

Implement the bindings for the Map struct based on the exposed bindings.

  • Create class definition
  • ymap
  • ymap_insert
    • Boolean
    • Double
    • Long
    • String
    • Bytes
    • Collection
    • Object
    • Null
    • Undefined
    • Text
    • Array
    • Map
    • Document
    • XML Element
    • XML Text
  • ymap_get
    • Boolean
    • Double
    • Long
    • String
    • Bytes
    • Collection
    • Object
    • Null
    • Undefined
    • Text
    • Array
    • Map
    • Document
    • XML Element
    • XML Text
    • Extras
      • Add tests to cover when getting a Map or Text might be null (like when the type under that key is already mapped to a different type).
  • ymap_remove
  • ymap_remove_all
  • ymap_len
  • ymap_iter
  • ymap_iter_next
  • ymap_iter_destroy
  • ymap_entry_destroy
  • ymap_observe
  • ymap_unobserve
  • ymap_event_keys
  • ymap_event_target
  • ymap_event_path

Extra:

  • Move *.Types.Map* declarations to *.Types.Maps.Map*
  • Release resources for Input operations
  • Release resources for Output operations
  • Release resources for MapEntry operations
  • Release resources for string operations

For each bound method, the implementation and unit tests must be written.

Create the Doc bindings

Implement the bindings for the Doc struct based on the exposed bindings.

  • Create class definition
  • ydoc_new
  • ydoc_clone
  • ydoc_destroy
  • ydoc_new_with_options
  • ydoc_id
  • ydoc_guid
  • ydoc_collection_id
  • ydoc_should_load
  • ydoc_auto_load
  • ydoc_observe_updates_v1
  • ydoc_unobserve_updates_v1
  • ydoc_observe_updates_v2
  • ydoc_unobserve_updates_v2
  • ydoc_observe_after_transaction
  • ydoc_unobserve_after_transaction
  • ydoc_clear
  • ydoc_observe_clear
  • ydoc_unobserve_clear
  • ydoc_read_transaction
  • ydoc_write_transaction
  • ydoc_load
  • ydoc_observe_subdocs
  • ydoc_unobserve_subdocs

For each bound method, the implementation and unit tests must be written.

Integrate new method ytransaction_alive

See

/// Check if current branch is still alive (returns `Y_TRUE`, otherwise `Y_FALSE`).
/// If it was deleted, this branch pointer is no longer a valid pointer and cannot be used to
/// execute any functions using it.
#[no_mangle]
pub unsafe extern "C" fn ytransaction_alive(txn: *const Transaction, branch: *mut Branch) -> u8 {
    if branch.is_null() {
        Y_FALSE
    } else {
        let txn = txn.as_ref().unwrap();
        let branch = branch.as_ref().unwrap();
        if txn.store().is_alive(&BranchPtr::from(branch)) {
            Y_TRUE
        } else {
            Y_FALSE
        }
    }
    ```

Create the XmlElement bindings

Implement the bindings for the XmlElement struct based on the exposed bindings.

  • Create class definition
  • yxmlelem
  • yxmlelem_tag
  • yxmlelem_string
  • yxmlelem_insert_attr
  • yxmlelem_remove_attr
  • yxmlelem_get_attr
  • yxmlelem_attr_iter
  • yxmlattr_iter_destroy
  • yxmlattr_iter_next
  • yxmlelem_child_len
  • yxmlelem_insert_text
  • yxmlelem_insert_elem
  • yxmlelem_remove_range
  • yxmlelem_get
  • yxml_next_sibling
  • yxml_prev_sibling
  • yxmlelem_parent
  • yxmlelem_first_child
  • yxmlelem_tree_walker
  • yxmlelem_tree_walker_destroy
  • yxmlelem_tree_walker_next
  • yxmlelem_observe
  • yxmlelem_unobserve
  • yxmlelem_event_target
  • yxmlelem_event_path
  • yxmlelem_event_delta
  • yxmlelem_event_keys
  • yobserve_deep
  • yunobserve_deep

For each bound method, the implementation and unit tests must be written.

Investigate possible memory leaks

In #18, there has been a mention of possible memory leaks in the operations executed through YDotNet.

This issue intends to:

  1. Investigate the possible issues
  2. Document the discoveries
  3. Create a procedure and, possibly, an automated way to detect such issues in the future

Investigated scenarios (this list is not final):

  • Doc
    • Create instances without options
    • Create instances with options
    • Read Id
    • Read Guid
    • Read CollectionId
    • Read ShouldLoad
    • Read AutoLoad
    • Invoke Clone() and Dispose()
    • Observe clear (leaking)
    • Observe updates V1 (leaking)
    • Observe updates V2 (leaking)
    • Observe after transaction (leaking)
    • Observe subdocs (leaking)
  • Text
    • Create instances
    • Insert text
    • Insert embed
    • Remove text
    • Format text
    • Read the chunks
    • Read the string representation
    • Read the length
    • Observe changes (leaking)
  • Array
    • Create instances
    • Insert range
    • Remove range
    • Get by index
    • Move by index
    • Iterate (leaking)
    • Read the length
    • Observe changes (leaking)
  • Map
    • Create instances
    • Insert
    • Read by key
    • Remove by key
    • Remove all
    • Iterate
    • Read length
    • Observe changes (leaking)
  • XmlText
    • Create instances
    • Insert text by index
    • Remove text by index
    • Format text by index
    • Insert embed by index
    • Insert attribute
    • Read attribute
    • Remove attribute
    • Iterate
    • Read as string
    • Read the previous sibling
    • Read the next sibling
    • Read the length
    • Observe changes (leaking)
  • XmlElement
    • Create instances
    • Read tag
    • Read as string
    • Insert attribute
    • Read attribute
    • Remove attribute
    • Iterate
    • Read child length
    • Insert text by index
    • Insert element by index
    • Remove items by index
    • Read item by index
    • Read the previous sibling
    • Read the next sibling
    • Read parent element
    • Read first child
    • Walk the tree
    • Observe changes (leaking)
  • Branch
    • Read a sticky index
    • Observe deep (leaking)
    • Create read transaction
    • Create write transaction

Other fixes:

  • Change subscriptions to be stored in their parent collections to prevent Delegate disposal

Implement automated tests pipeline

Add the necessary configuration files to run tests for all supported platforms:

  1. When pushing changes to main
  2. When creating PRs (to make sure the changes don't break features)

Take inspiration from previous work done here and here.

Create XmlText bindings

Implement the bindings for the XmlText struct based on the exposed bindings.

  • Create class definition
  • yxmltext
  • yxmltext_insert
  • yxmltext_len
  • yxmltext_string
  • yxmltext_insert_embed
  • yxmltext_remove_range
  • yxmltext_format
  • yxmltext_insert_attr
  • yxmltext_get_attr
  • yxmltext_remove_attr
  • yxmltext_attr_iter
  • yxml_next_sibling
  • yxml_prev_sibling
  • yxmltext_observe
  • yxmltext_unobserve
  • yxmltext_event_target
  • yxmltext_event_path
  • yxmltext_event_keys
  • yxmltext_event_delta
  • yobserve_deep
  • yunobserve_deep

For each bound method, the implementation and unit tests must be written.

Create the Transaction bindings

Implement the bindings for the Transaction struct based on the exposed bindings.

  • Create class definition
  • ytransaction_subdocs
  • ytransaction_commit
  • ytransaction_writeable
  • ytransaction_state_vector_v1
  • ytransaction_state_diff_v1
  • ytransaction_state_diff_v2
  • ytransaction_apply
  • ytransaction_apply_v2
  • ytransaction_snapshot
  • ytransaction_encode_state_from_snapshot_v1
  • ytransaction_encode_state_from_snapshot_v2

For each bound method, the implementation and unit tests must be written.

Fix Publish workflow

Tasks:

  • Add NuGet API key to publish new package versions from this repository
  • Add README to the main project (YDotNet)
  • Fix typos in the NuGet package description
  • Fix authors list to include goldsam
  • Bump version to 0.2.10

Package yrs binaries as Nuget packages

Consumption of this work (upon release) would be easier if the Yrs binaries were provided as nuget packages. If you would like, I could put together a PR for this.

Create the StickyIndex bindings

Implement the bindings for the StickyIndex struct based on the exposed bindings.

  • Create class definition
  • ysticky_index_destroy
  • ysticky_index_assoc
  • ysticky_index_from_index
  • ysticky_index_read
  • ysticky_index_encode
  • ysticky_index_decode

For each bound method, the implementation and unit tests must be written.

ReferenceAccessor should be a factory

The reference accessor is actually super weird. First, an object is created. Then we check the handle and discard this object again. This puts unnecessary pressure on the GC.

I think we should just rename it to ReferenceFactory and refactor it to methods like this:

public static Output? Output(nint handle, bool shouldDispose)
{
     return handle == nint.Zero ? null : new Output(handle, shouldDispose);
}

Depending on #48 it might event be obsolete.

Please assign me, if you agree. Happy to provide a PR.

WebSocket Client

I think it would be great to have a client class, in my case, just for testing the server.

Planned features

Nice project.

What is your planned scope? Are you also planning to publish ready to use packages for signalr or something that can easily be integrated into the server? Do you need help?

I am planning to integrate it into https://github.com/squidex/squidex

Add README to the repository

Add an initial version of a README file in the repository to present it to visitors.

Take inspiration and mention y-crdt and other language binding projects (like ypy and yrb).

Fix unit tests

The unit tests in the project were broken because the binaries used were built from y-crdt/y-crdt instead of LSViana/y-crdt, then it was missing some fixes (mentioned here).

@LSViana fixed the issue by building the binaries from LSViana/y-crdt again. But this broke the unit tests differently now (as seen here).

They should be fixed before the repository is moved into the y-crdt organization.

youtput_read_json_undefined isn't an exported yffi symbol

@LSViana I'm getting the following error against v0.16.10 of yrs:

System.EntryPointNotFoundException : Unable to find an entry point named 'youtput_read_json_undefined' in DLL 'yrs.dll'.

Are you using a modified version of the library? I'm not finding that symbol in any branch.

YDotNet stops persisting to storage after 1st invalidation

Hey, I noticed some odd behavior when trying to work with Redis storage (I believe this also impacts InMemoryStorage).

During the first 10 seconds, while the cache is valid, I can see several writes to Redis. But then, once the cache gets invalidated and we initialize a new Doc, it stops working.

You can find a working example here: https://github.com/vdurante/ydotnet/tree/redis-write-bug

Using your demo project, this is what I did:

Set storage type in appsettings.Development.json:

"Type": "Redis",

Modify initialization of redis storage:

yDotNet.AddRedisStorage(
    opts =>
    {
        opts.Expiration = _ => TimeSpan.FromMinutes(5);
    });
yDotNet.AddRedis(options =>
{
    options.Configuration =
        ConfigurationOptions.Parse(
            builder.Configuration["Storage:Redis:ConnectionString"]!); // this value is wrong in the demo project. I had to rename to Storage:Redis:ConnectionString
});

Add this to Program.cs:

builder.Services.PostConfigure<DocumentManagerOptions>(
  dmo =>
  {
      dmo.CacheDuration = TimeSpan.FromSeconds(10);
  });

Added the following to StoreDocAsync in RedisDocumentStorage.cs:

Console.WriteLine("size: " + doc.Length);

NullReferenceException in Commit

Hi,

I am testing a scenario where I want to push a notification to another document when a new comment has been created.

https://github.com/SebastianStehle/ydotnet/blob/main/Demo/Callback.cs#L50

But I get a NullReferenceException in the commit method and the following error:

thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: BorrowMutError', yffi\src\lib.rs:466:10
stack backtrace:
   0: rust_begin_unwind
             at /rustc/d5c2e9c342b358556da91d61ed4133f6f50fc0c3/library\std\src/panicking.rs:593:5
   1: core::panicking::panic_fmt
             at /rustc/d5c2e9c342b358556da91d61ed4133f6f50fc0c3/library\core\src/panicking.rs:67:14
   2: core::result::unwrap_failed
             at /rustc/d5c2e9c342b358556da91d61ed4133f6f50fc0c3/library\core\src/result.rs:1651:5
   3: core::result::Result<T,E>::unwrap
             at /rustc/d5c2e9c342b358556da91d61ed4133f6f50fc0c3\library\core\src/result.rs:1076:23
   4: ydoc_observe_updates_v1
             at D:\_other\y-crdt\yffi\src\lib.rs:460:20

Have you seen this before? I have no idea about rust and cannot narrow it down yet.

v0.2.13 release?

Hi! I've been trying to get this running on my M1 Mac (Apple Silicon, ARM).

I had errors loading the dylib, which happily looks like to be fixed in #81 and then the version bumped in #82 to 0.2.13. However, the nuget feed still only has v0.2.12.

Could you create a new release? Thanks!

Create the UndoManager bindings

Implement the bindings for the UndoManager struct based on the exposed bindings.

  • Create class definition
  • yundo_manager
  • yundo_manager_destroy
  • yundo_manager_observe_added
  • yundo_manager_unobserve_added
  • yundo_manager_undo
  • yundo_manager_redo
  • yundo_manager_can_undo
  • yundo_manager_can_redo
  • yundo_manager_clear
  • yundo_manager_stop
  • yundo_manager_observe_popped
  • yundo_manager_unobserve_popped
  • yundo_manager_add_scope
  • yundo_manager_add_origin
  • yundo_manager_remove_origin

For each bound method, the implementation and unit tests must be written.

Improve Output cell

Improve issues mentioned at #44:

  • Add the Type property to the Output cell
  • Possibly remove nullable return types and check type before reading the value

Basic architecture and thoughts

Basic Plan

I have started to play around with the code base a little bit. My idea would be the following:

  1. Create a server project that provides a network agnostic business layer and handles the following:
  • Loading documents
  • Saving documents (configurable)
  • Invoking hooks when something has changed (I need that).
  • Provide some methods to update stuff from server side (I also need that).
  1. Create network layers on top of the that like SignalR or WebSocket.

Questions

What I don't understand is:

Initial sync process

I am talking about the following code: https://github.com/yjs/ycs/blob/main/samples/YcsSample/Yjs/YcsManager.cs#L109. Perhaps I have some issues in understanding the general yjs protocol.

For my understanding, the client calls GetMissing and actually sends his document to the server, which then makes a diff and then gets a diff from the client and the server document.

After that the client would receive updates. But afaik this only works because the global lock. Otherwise The client could loose update messages. So would it not make more sense to send all update messages to the client and let him sort out that problem?

Scaling

I had a look to the following code: https://github.com/ueberdosis/hocuspocus/blob/main/packages/extension-redis/src/Redis.ts

So if I understand it properly, there are 2 concepts:

  1. The document potentially exists on all servers.
  2. The server documents and the awareness information are shared with pubsub.

Then signalr would not really make sense I think.

Exception calling Get* on top level field without first defining it

Consider this unit test based on the example in the root README.md. Note that on remoteDoc I never call remoteDoc.Text("name") to define the field.

[Fact]
public void ReadingUndeclaredPropertyReturnsNull()
{
    Doc localDoc = new Doc();
    Text localText = localDoc.Text("name");

    Transaction localTransaction = localDoc.WriteTransaction();
    localText.Insert(localTransaction, 0, "Y-CRDT");
    localTransaction.Commit();

    // Set up the remote document.
    Doc remoteDoc = new Doc();
    // remoteDoc.Text("name"); 
    // ^^^^^^^^^^^^^^^^^^^^^^. NOTE: explicitly not declaring the name field

    // Get the remote document state vector.
    Transaction remoteTransaction = remoteDoc.WriteTransaction();
    byte[] remoteState = remoteTransaction.StateVectorV1();

    // Calculate the state diff between the local and the remote document.
    localTransaction = localDoc.ReadTransaction();
    byte[] stateDiff = localTransaction.StateDiffV1(remoteState);
    localTransaction.Commit();

    // Apply the state diff to synchronize the remote document with the local changes.
    TransactionUpdateResult result = remoteTransaction.ApplyV1(stateDiff);
    remoteTransaction.Commit();

   // at this point, remoteDoc should have the "name" field since it synced with localDoc

    using (remoteTransaction = remoteDoc.ReadTransaction())
    {
        Text? textProperty = remoteTransaction.GetText("name");
        // as a new user, I expected this to return the Text since the sync should have created
        // it.

        // Instead, the docs state it should be null: "Returns the Text at the Doc root level, identified by name, 
        // or null if no entry was defined under name before." However, it throws instead.
        Assert.Null(textProperty); 
    }
}

I had expected the call to remoteTransaction.GetText("name") to return the Text object since
by virtue of syncing with localDocument the property does now exist. Barring that, I expected it to return null per the documentation of the GetText method: "Returns the Text at the Doc root level, identified by name, or null if no entry was defined under name before."

In reality, it threw an exception:

YDotNet.YDotNetException: Expected 'Text', got '0'.

YDotNet.YDotNetException
Expected 'Text', got '0'.
   at YDotNet.Document.Transactions.Transaction.GetWithKind(String name, BranchKind expectedKind)
   at YDotNet.Document.Transactions.Transaction.GetText(String name)
   at Ballpark.Tests.DocTests.ReadingUndeclaredPropertyReturnsNull() in /Users/mattburke/projects/ballpark/server/Ballpark.Tests/DocTests.cs:line 57
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
  • Is this expected behavior? Is a documentation update in order?
  • If throwing is correct, it might be useful to add to the exception message, something like "Did you forget to define the field?".

I would love to contribute an improvement if there's a direction you prefer to take!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.