ralfbiedert / interoptopus Goto Github PK

I am using Interoptopus to generate C# bindings for Unity.
This is my Generator:

    Generator::new(config, rust_lib::my_inventory())
        //.add_overload_writer(DotNet::new())
        .add_overload_writer(Unity::new())
        .write_file("bindings/csharp/Interop.cs")?;

I am trying to pass a mutable slice from C# to Rust and write into it like so:

#[ffi_function]
#[no_mangle]
pub extern "C" fn mutate_slice_u32(slice: &mut FFISliceMut<u32>) {
    let data = vec![1, 2, 3, 4, 5];
    for (idx, el) in data.iter().enumerate() {
        slice[idx] = *el;
    }
}

The relevant generated binding looks like this:

        [DllImport(NativeLib, CallingConvention = CallingConvention.Cdecl, EntryPoint = "mutate_slice_u32")]
        public static extern void mutate_slice_u32(ref SliceMutu32 slice);

        #if UNITY_2018_1_OR_NEWER
        public static void mutate_slice_u32(NativeArray<uint> slice)
        {
            var slice_slice = new SliceMutu32(slice);
            mutate_slice_u32(ref slice_slice);;
        }
        #endif

Issue is I am getting an error in Unity saying:
Assets/Plugins/Interop.cs(53,35): error CS1729: 'SliceMutu32' does not contain a constructor that takes 1 arguments

Am I doing anything especially crazy here?

Otherwise users can safely provide bad type info at the FFI boundary.

Hm.

On the other hand, it doesn't solve the problem that a surrogate attribute can just use the wrong surrogate function ...

Hi, first of all thanks for the great library! I am able to return a AsciiPointer from a method of my service, but when I try to return a AsciiPointer nested in a struct I get a runtime error:

System.Runtime.InteropServices.MarshalDirectiveException: Method's type signature is not PInvoke compatible.
   at My.Company.Interop.counter_service_nested_string(IntPtr context)
   at My.Company.CounterService.NestedString() in C:\Users\nico\tmp\lib_test\bindgen\generated\Interop.cs:line 288
   at ConsoleApplication42.Program.Main(String[] args) in C:\Users\nico\source\tmp\ConsoleApplication42\ConsoleApplication42\Program.cs:line 51

edit: example is here: https://github.com/njust/interoptopus_test/blob/master/lib_test/src/lib.rs

The following changes I'd like to land in 0.15, help wanted

• Infer prefix in #[ffi_service] and make it optional
• Add sample how to use feature flags to disable Interoptopus unless generating bindings
• Improve the surrogate type situation, e.g., by instead relying on transmutable surrogates. Needs some thought.
• Maybe struct methods #29 (depends on how much work they are)
• Maybe implement #27, or rule out why they are a bad idea
• Make sure void service methods don't need annotations

When debugging generated code with debug flags the output is sometimes hard to read on the console. Also review what can be done to make macro development easier and / or existing options more discoverable, compare #90 .

I am new to rust ，c# string encoding is unicode and rust is uft-8. but I don't know how to handle them.
I want to modify string in rust side, and read it in c#

rust code

use std::ffi::{CStr, CString};
use std::fmt::{Display, Formatter, Pointer};
use interoptopus::patterns::result::FFIError;
use interoptopus::patterns::slice::FFISlice;
use interoptopus::{
    ffi_function, ffi_service, ffi_type, function, pattern, Inventory, InventoryBuilder,
};
use interoptopus::{ffi_service_ctor, ffi_service_method};
use libc::c_char;

....

#[ffi_type]
#[repr(C)]
pub struct FLText {
    pub id: u64,
    pub text: *const c_char,
}

#[ffi_function]
pub  extern "C"  fn test_return_str(mut fltext:FLText) -> FLText {
    let c_str = unsafe {
        assert!(!fltext.text.is_null());
        CStr::from_ptr(fltext.text)
    };
    let r_str = c_str.to_str().unwrap().to_string();
    let r_str=r_str.as_bytes();
    // let a_unicode='の'.escape_unicode();
    fltext.text = CString::new(r_str).unwrap().into_raw();
    fltext
}

c# code

                var flText = new FLText();
                string flStr = "你好";
                flText.id = 123123;
                flText.text = GCHandle.Alloc(flStr.ToArray(), GCHandleType.Pinned).AddrOfPinnedObject();
                var result = x.TestReturnStr(flText);
                var resultStr = Marshal.PtrToStringAuto(result.text);

Thanks for advice~

Dataful enums are a powerful feature of rust's type system, and unfortunately this means they are difficult to represent elegantly in many other languages. However, at minimum dataful enums in rust can be represented as a tagged union in C, and in languages with structural inheritance, this could be translated to a class or record type hierarchy.
Adding support for dataful enums, even rudimentary support only mapping them to tagged unions in the target languages, would be a very valuable feature for rust interoperability.

As an example of a simple dataful enum on rust side:

#[repr(u32,C)]
pub enum Value {
    VByte(u8),
    VFloat(f32),
}

And its equivalent on the C# side, including the facility to perform basic matching on the variant:

public struct Value
{
    private enum Tag
    {
        VByte,
        VFloat,
    }

    [FieldOffset(0)] private Tag tag;
    [FieldOffset(4)] private Byte byte_data;
    [FieldOffset(4)] private Single float_data;

    public void Match(Action<byte> byte_f, Action<Single> float_f)
    {
        Match<object>(b =>
        {
            byte_f(b); return null;
        },
        f =>
        {
            float_f(f); return null;
        });
    }
    public R Match<R>(Func<byte, R> byte_f, Func<Single, R> float_f)
    {
        switch (tag)
        {
            case Tag.VByte:
                return byte_f(byte_data);
            case Tag.VFloat:
                return float_f(float_data);
            default:
                throw new ArgumentOutOfRangeException();
        }
    }
}

Optionally, this tagged union could be converted to a class hierarchy, if this is deemed valuable:

public abstract class ValueClass
{
    public static ValueClass From(Value v)
    {
        return v.Match<ValueClass>(
            b => new ByteValue(b),
            f => new FloatValue(f)
        );
    }
    public class ByteValue : ValueClass
    {
        public Byte b;

        public ByteValue(byte b)
        {
            this.b = b;
        }
    }
    public class FloatValue : ValueClass
    {
        public Single f;

        public FloatValue(float f)
        {
            this.f = f;
        }
    }
}

Would be useful to have an option in C# code generation config to customize access modifiers on generated classes/structs. In my usage of interoptopus, I'm creating a friendlier C# wrapper around a native DLL so would prefer not to expose the generated classes in the wrapper library interface. Currently I'm doing a regex find and replace on the file after generation to change the accessors to 'internal'.

I feel like there's no good way to access all of (instead of indexer (which copies data anyway)) the slice data in c# right now without copying it.

I propose adding a method to the c# generated version of FFISlice and FFISliceMut that has a ReadOnlySpan and Span respectively.

For example:

///A pointer to an array of data someone else owns which may be modified.
[Serializable]
[StructLayout(LayoutKind.Sequential)]
public partial struct SliceMutf32
{
    ///Pointer to start of mutable data.
    IntPtr data;
    ///Number of elements.
    ulong len;
}

public partial struct SliceMutf32
{
    public Span<float> AsSpan()
    {
        unsafe
        {
            return new Span<float>((float*)data, (int)len);
        }
    }
}

Not sure how you'd do it without unsafe though.

Should probably also make it optional, since Span is part of System.Memory which isn't included in the standard framework on all .net versions.

To my pleasant surprise, I discovered that the AsciiPointer pattern plainly uses Rust CString, which is UTF-8 instead of ASCII. This is not an issue when transferring strings into Rust, as ASCII is a subset of UTF-8, so Rust will happily handle both. However, it might surprise users when accepting strings from Rust, when the Rust code accidentally outputs unicode characters (there's currently no check on this).

There seem to be two solutions to me: either perform a check in the Rust-implementation of AsciiPointer to make sure the string actually is ASCII, or, for me preferable, just support the unicode. For C both are null-terminated byte-strings anyway, so it just needs a clear warning sign for developers. For C# I've made a few small changes to the backend to explicitly marshal strings as UTF-8 which seems to work quite nicely (see https://github.com/seaeagle1/interoptopus/tree/utf8 ). I don't know much Python, but I'm sure it's also able to handle unicode strings.

Thank you for this awesome library!

It was really easy to get started with the provided examples, but I'm having hard time to return String (AsciiPointer)
more than once from Rust to CS via the Service pattern.

With the following snippet I managed to return the string once, but the second call is causing the program to crash:

use crate::result::{Error, FFIError};
use interoptopus::{ffi_service, ffi_service_ctor, ffi_service_method, ffi_type};
use interoptopus::patterns::string::AsciiPointer;
use std::ffi::CString;

#[ffi_type(opaque)]
#[derive(Default)]
pub struct Sample {
    pub number: usize,
    name: CString,
}


#[ffi_service(error = "FFIError", prefix = "sample_")]
impl Sample {
    #[ffi_service_ctor]
    pub fn new_with(number: u32) -> Result<Self, Error> {
        Ok(Self {
            number: number as usize,
            name: CString::new(format!("HELLO_{}", number)).unwrap(),
        })
    }

    #[ffi_service_method(direct)]
    #[allow(non_snake_case)]
    pub fn Name(&self) -> AsciiPointer {
        // FIXME: Name can only be called once. second invocation causes crash.
        AsciiPointer::from_cstr(&self.name)
    }
}

interoptopus::inventory!(
    my_inventory, 
    [], 
    [], 
    [], 
    [Sample]);

My theory is that the Sample.CString gets freed after AsciiPointer::from_cstr is returned.
I tried to solve this by introducing new lifetime for name, but didn't succeed: always ended up having obscure
error coming from the ffi_service macro.

Any suggestion / help would be greatly appreciated!

Currently the C# backend generates the following if the API version check pattern is implemented and the API version check fails:

throw new Exception($"API reports hash {api_version} which differs from hash in bindings (8493736274713624474). You probably forgot to update / copy either the bindings or the library.");

It is recommend practice to never throw an Exception of the base 'Exception' type:

https://docs.microsoft.com/en-us/dotnet/standard/design-guidelines/using-standard-exception-types.

There's not an incredibly obvious answer as to what system exception type to use instead, potentially a 'TypeLoadException'? Alternatively a custom generated Exception type could be created and used.

In order to use extern functions in burst, structs have to be passed by reference. If I try changing all my slices to be &mut slices, the extern functions are generated nicely with out SliceMut but I lose the other functions that are normally generated with them. (The ones with the pinning of arrays and conversion to SliceMut. If I want those helper functions for both Slice and &mut Slice does this mean that I need to add a pattern for handling the &mut version of a Slice the same way that the Slice is handled to the C# generator?
As an aside I would also like the ability to generate slices from native arrays. I see that you've already got some Unity specific flagging, would you be interested in PRs to add some more Unity specific interoperability to the generated c# behind those flags?

Examples:

• Pointers to arrays
• Multidimensional arrays

If issues are found code generation should also be cleaned accordingly.

I clone interoptopus, remove examples/hello_world/bindings, then cargo test. Following errors are generated,

     Running tests/bindings.rs (/home/ccwu/Projects/botnana/interoptopus/target/debug/deps/bindings-ec8e39723c19b389)

running 3 tests
Error: IO(Os { code: 2, kind: NotFound, message: "No such file or directory" })
Error: IO(Os { code: 2, kind: NotFound, message: "No such file or directory" })
Error: IO(Os { code: 2, kind: NotFound, message: "No such file or directory" })
test bindings_cpython_cffi ... FAILED
test bindings_csharp ... FAILED
test bindings_c ... FAILED

failures:

---- bindings_cpython_cffi stdout ----
thread 'bindings_cpython_cffi' panicked at 'assertion failed: `(left == right)`
  left: `1`,
 right: `0`: the test returned a termination value with a non-zero status code (1) which indicates a failure', /rustc/897e37553bba8b42751c67658967889d11ecd120/library/test/src/lib.rs:184:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- bindings_csharp stdout ----
thread 'bindings_csharp' panicked at 'assertion failed: `(left == right)`
  left: `1`,
 right: `0`: the test returned a termination value with a non-zero status code (1) which indicates a failure', /rustc/897e37553bba8b42751c67658967889d11ecd120/library/test/src/lib.rs:184:5

---- bindings_c stdout ----
thread 'bindings_c' panicked at 'assertion failed: `(left == right)`
  left: `1`,
 right: `0`: the test returned a termination value with a non-zero status code (1) which indicates a failure', /rustc/897e37553bba8b42751c67658967889d11ecd120/library/test/src/lib.rs:184:5


failures:
    bindings_c
    bindings_cpython_cffi
    bindings_csharp

test result: FAILED. 0 passed; 3 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

error: test failed, to rerun pass `--test bindings`

I'm trying to create C# binding for libsignal. They already have ffi with cbindgen.
Is it possible to reuse it, without going through libsignal code and apply changes for Interoptopus?

Hello,

I'm trying to create a rust service which returns a Vec<f64> / List<double> to c#.
Unfortunately I found no examples in this direction and my attempts have failed:

#[ffi_service(error = "FFIError", prefix = "FOO_")]
impl FOO {
    ...

    #[ffi_service_method(wrap="direct")]
    pub fn results(&self) -> FFISlice<f64> {
        let slice = vec!{1.12, 3.14};
        FFISlice::from_slice(&slice)
    }
}

The above code results the following compile time error:

error[E0599]: no function or associated item named `default` found for struct `FFISlice<'_, f64>` in the current scope
  --> src/foo.rs:18:1
   |
18 | #[ffi_service(error = "FFIError", prefix = "FOO_")]
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ function or associated item not found in `FFISlice<'_, f64>`
   |
   = note: this error originates in the attribute macro `ffi_service` (in Nightly builds, run with -Z macro-backtrace for more info)

This might not be an actual issue, just my lack of understanding of how interoptopus works.

Any suggestion / help /pointer would be greatly appreciated!

Support types (esp. structs) that use const generics.

In C all strings are terminated by a null byte (0x0), which means that strings using the idiomatic C representation cannot contain null.
However, many languages, including Python and C# support strings that store length as a separate value and can contain null. It would be nice if Interoptopus provided a canonical way to pass such strings accross the FFI boundary.

The only way to do this currently that I can figure out is to use a slice of bytes and convert manually whenever you cross the FFI boundary, which is incovenient and in for example Python requires copying the entire string for every conversion.

I had a branch with some progress on this before a small "SSD upgrade accident", basic idea was struct methods needed to be defined as traits, with annotations on traits governing their forward implementation, as that was the (IIRC) the only way to get reasonable (implementation and UX) proc-macro support.

Curious as to why the generated bindings use GCHandles instead of just passing a pointer to the slice inside a fixed statement

• Extract common code into functions
• Better support types / patterns in parameters, rvals and fields that currently panic for no good reason

Right now C# is the only backend supporting all patterns. Python should be on par.

• Check all patterns in interoptopus::patterns
• Implement each missing pattern generation in Python

Compare #83

Reasons:

• Forcing our users' users to install cffi feels clumsy for an API generator.
• Posts from some years ago usually stated cffi is faster than ctypes. After some naive benchmarking (e.g., calling the reference project's primitive_i8(x) 1M times and measuring time) in Python 3.9 I can't confirm that. Call times were ~400ns per invocation with cffi, and about ~350ns using ctypes.
• Ctypes code will probably look cleaner w/o having to parse embedded header and creating wrapper-wrappers.

Hi!

I am sure this isn't an actual issue but rather me not understanding how things work.
I am trying give a slice of strings from Rust to C# but can't figure out how to structure the flow.

I've been trying to do something like this (my assumption is Rust should own the underlying CString and then just return an AsciiPointer to it)...

#[ffi_type(opaque)]
pub struct FFIPlayerStruct {
    c_string: CString,
    id: u64
}

#[ffi_service(error = "MyFFIError", prefix = "ffi_player_struct_")]
impl FFIPlayerStruct {

/// omitted constructor

    #[ffi_service_method(on_panic = "return_default")]
    pub fn get_player_name(&self) -> AsciiPointer{
        AsciiPointer::from_cstr(&self.c_string)
    }
}

Some other service holds a vector of player structs. My understanding is that passing a mutable slice with pre-allocated size is how we're meant to pass a vector/array.

    pub fn get_player_structs(&self, buffer: &mut FFISliceMut<FFIPlayerStruct>)  -> Result<(), Error> {
        for (idx, player_struct) in self.player_structs.iter().enumerate() {
            buffer[idx] = FFIPlayerStruct { c_string: player_struct.c_string.clone(), id: player_struct.id }
        }
        Ok(())
    }

The interop bindings generated look like this (I assume the IntPtr is because FFIPlayerStruct is opaque)

        public SliceMutFFIPlayerStruct(NativeArray<IntPtr> handle)
        {
            unsafe
            {
                this.data = new IntPtr(NativeArrayUnsafeUtility.GetUnsafeReadOnlyPtr(handle));
                this.len = (ulong) handle.Length;
            }
        }

My question is if this is the right way to go around the problem and if so - how are you meant to call get_player_name() from an IntPtr? Apologies but my C# isn't the strongest.

Btw, been diving deeper and deeper into the crate over the last few weeks and it's really great!

I've been looking around for this option and I don't think it exists. I'm basically looking to use types in function signatures like Vec3 or Mat4 which I already have full implementations of on the target language. I want the named types emitted in function signatures, but I don't want them declared on their own.
I'm happy to take a look at implementing this, I think it would make the most sense as an attribute on ffi-type

Merge redundant code in the C# backend, esp. w.r.t.:

• overloads
• pattern helpers

I don't currently have the time to look into creating this myself, but it would be nice to have a NodeJS backend, since as far as I can find no similar project currently supports both Python and NodeJS.

Not sure if i'm doing something wrong, but i'm fairly sure i'm not...

If you have an FFISlice parameter in a function it seems to point to garbage data or something on the rust side. But moving the same function to a service makes it work like expected.

I made a project as an example:

https://github.com/Skippeh/interoptopus_slice_issue

Outputs:

It seems to behave the same on 0.13.15. Haven't tried earlier versions.

I have only tried C# but i think the problem is in the rust code.

Developers and myself often have trouble running unit tests, compare #90. It would be nice to rework how testing works (e.g., how python and dotnet are handled, how expected output is matched, file & folder structure of test projects, ...)

When Config::rename_symbols is true, the generated API version check does not rename the call to the API version function, resulting in an uncompilable C# source file.

The CSharpWriter::write_abi_guard function likely needs to call self.converter().function_name_to_csharp_name(function, self.config().rename_symbols) in a similar way to CSharpWriter::write_function_declaration.

For instance in rust we have

#[ffi_type]
#[repr(C)]
pub struct Vec2 {
    pub x: f32,
    pub y: f32,
}

#[ffi_function]
#[no_mangle]
pub extern "C" fn my_function(input: Vec2) -> Vec2 {
    input
}

I'd like to use UnityEngine.Vector2 instead of generated Vec2 type.

        /// Function using the type.
        [DllImport(NativeLib, CallingConvention = CallingConvention.Cdecl, EntryPoint = "my_function")]
        public static extern UnityEngine.Vector2 my_function(UnityEngine.Vector2 input);

Is this currently possible?

Currently c_char type parameters get transformed to int8_t in the C backend. This results in all FFI function calls accepting a C string as input (*const c_char) to generate header files which express this in a way that requires a cast to *int8_t to use. Ideally the C backend should transform c_char type into char.

Happy to look at this myself proving there's no good reasons against this behavior?

Somewhat experimental idea, needs validation, also see #68:

Names tbd:

• A generic FFIPointer<T, SingleElementOrMultiple, RustRW, FFIRW> indicating how that pointer is being used (type T of target, single or multiple elements, and who reads and writes; bonus points if we can make T so that it also accepts and enum of type alternatives although I wouldn't know how to do that)
• various aliases to common FFIPointer<A, B, C, D> combinations.
• The generic FFIPointer would fallback to *T or *c_void in backends for now or could be used for other optimizations (e.g., deciding if an #[In, out] should be applied.
• bonus-bonus points if MultipleElements can also encode variable or constant many elements.

This pointer could also subsume Callbacks with the help of const generics, e.g., Callback<Name="Bla"> once &str are available.

Main questions to resolve are:

• Can this be implemented?
• Will this make writing APIs more ergonomic (e.g., better C# bindings w.r.t IntPtr, #[In/Out], ...)?
• Will this be cleaner or more messy than having multiple independent types (e.g., w.r.t. our maintenance)?

If you look at the generated IEnumerable code, it generates a plain yield return enumerator. That means there's an allocation on every GetEnumerator call - which is normally avoided by providing a concrete StructEnumerator GetEnumerator() overload. The code seems to do a lot of redundant work (exception is checked on the indexer hot path, so indexer will never be inlined).

Would it make a bit more sense to return a ReadOnlySpan<T> view over the data? That would also give a bit of flexibility to the caller at no performance expense. That would also remove the inheritance information from the struct,s.

Is it possible to convert rust document comments into C# style so we can generate C# API document with it?

for instance:

/// This function has documentation.
#[ffi_function]
#[no_mangle]
pub extern "C" fn documented(_x: StructDocumented) -> EnumDocumented {
    EnumDocumented::A
}

should be convert to:

/// <summary>
/// This function has documentation.
/// </summary>
...

I can make a new issue if you'd prefer, but just a headsup. Any IntPtrs on structs generated by this need to be marked with [NativeDisableUnsafePtrRestriction] in order to be used with burst stuff (it's a runtime enforced exception)

[Serializable]
    [StructLayout(LayoutKind.Sequential)]
    public partial struct SliceMutBvhNode
    {
        [NativeDisableUnsafePtrRestriction]
        IntPtr data;
        ulong len;
    }

Originally posted by @dbenson24 in #17 (comment)

Single field #[repr(C)] structs containing integer fields should generate these helpers:

    public partial struct InstanceID : IComparable<InstanceID>, IEquatable<InstanceID>
    {
        public static InstanceID From(int id)
        {
            return new InstanceID { x0 = (ulong)id };
        }

        public int CompareTo(InstanceID other)
        {
            return x0.CompareTo(other.x0);
        }

        public override string ToString()
        {
            return x0.ToString();
        }

        public override int GetHashCode()
        {
            return x0.GetHashCode();
        }

        public bool Equals(InstanceID other)
        {
            return x0 == other.x0;
        }
    }

I wanted to bump my api version after some internal bugfixes (without changing the API surface), and was confused when changing the APIVersion returned by my api_guard function did not change the bindings generated.

Here is an example of what I expected would update the version embedded in the (C#) bindings.

pub extern "C" fn ffi_api_guard() -> interoptopus::patterns::api_guard::APIVersion {
    let impl_version = 1;
    let api_version = interoptopus::patterns::api_guard::inventory_hash(&ffi_inventory());
    interoptopus::patterns::api_guard::APIVersion::new(api_version + impl_version)
}

It seems that the assumption that the APIVersion is always the inventory hash is made in the writer. I think it would make more sense, given that the api_guard function is developer defined, to use it's return value when generating the bindings.

As a workaround, I am just including a doc comment with "impl version: 1" on the ffi_api_guard function, and bumping that as needed to regenerate the bindings.