Code Monkey home page Code Monkey logo

protocol's Introduction

protocol

Build Status Crates.io MIT licensed

Documentation

Easy protocol definitions in Rust.

This crate adds a custom derive that can be added to types, allowing structured data to be sent and received from any IO stream.

Networking is built-in, with special support for TCP and UDP.

The protocol you define can be used outside of networking too - see the Parcel::from_raw_bytes and Parcel::raw_bytes methods.

This crate also provides:

  • TCP and UDP modules for easy sending and receiving of Parcels
  • A generic middleware library for automatic transformation of sent/received data
    • Middleware has already been written to support compression
    • Custom middleware can be implemented via a trait with two methods

Checkout the examples folder for usage.

Usage

Add this to your Cargo.toml:

[dependencies]
protocol = { version = "3.4", features = ["derive"] }

And then define a type with the #[derive(protocol::Protocol)] attribute:

#[derive(protocol::Protocol)]
struct Hello {
    pub a: String,
    pub b: u32,
}

Under the hood

The most interesting part here is the protocol::Parcel trait. Any type that implements this trait can then be serialized to and from a byte stream. All primitive types, standard collections, tuples, and arrays implement this trait.

This crate becomes particularly useful when you define your own Parcel types. You can use #[derive(protocol::Protocol)] to do this. Note that in order for a type to implement Parcel, it must also implement Clone, Debug, and PartialEq.

#[derive(Parcel, Clone, Debug, PartialEq)]
pub struct Health(f32);

#[derive(Parcel, Clone, Debug, PartialEq)]
pub struct SetPlayerPosition {
    pub position: (f32, f32),
    pub health: Health,
    pub values: Vec<String>,
}

Custom derive

Any user-defined type can have the Parcel trait automatically derived.

Example

#[derive(protocol::Protocol, Clone, Debug, PartialEq)]
pub struct Handshake;

#[derive(protocol::Protocol, Clone, Debug, PartialEq)]
pub struct Hello {
    id: i64,
    data: Vec<u8>,
}

#[derive(protocol::Protocol, Clone, Debug, PartialEq)]
pub struct Goodbye {
    id: i64,
    reason: String,
}

#[derive(protocol::Protocol, Clone, Debug, PartialEq)]
pub struct Node {
    name: String,
    enabled: bool
}

#[protocol(discriminant = "integer")]
#[derive(protocol::Protocol, Clone, Debug, PartialEq)]
pub enum PacketKind {
    #[protocol(discriminator(0x00))]
    Handshake(Handshake),
    #[protocol(discriminator(0xaa))]
    Hello(Hello),
    #[protocol(discriminator(0xaf))]
    Goodbye(Goodbye),
}

fn main() {
    use std::net::TcpStream;

    let stream = TcpStream::connect("127.0.0.1:34254").unwrap();
    let mut connection = protocol::wire::stream::Connection::new(stream, protocol::wire::middleware::pipeline::default());

    connection.send_packet(&Packet::Handshake(Handshake)).unwrap();
    connection.send_packet(&Packet::Hello(Hello { id: 0, data: vec![ 55 ]})).unwrap();
    connection.send_packet(&Packet::Goodbye(Goodbye { id: 0, reason: "leaving".to_string() })).unwrap();

    loop {
        if let Some(response) = connection.receive_packet().unwrap() {
            println!("{:?}", response);
            break;
        }
    }
}

Enums

Discriminators

Enum values can be transmitted either by their 1-based variant index, or by transmitting the string name of each variant.

NOTE: The default behaviour is to use the variant name as a string (string).

This behaviour can be changed by the #[protocol(discriminant = "<type>")] attribute.

Supported discriminant types:

  • string (default)
    • This transmits the enum variant name as the over-the-wire discriminant
    • This uses more bytes per message, but it very flexible
  • integer
    • This transmits the 1-based variant number as the over-the-wire discriminant
    • If enum variants have explicit discriminators, the
    • Enum variants cannot be reordered in the source without breaking the protocol
#[derive(protocol::Protocol, Clone, Debug, PartialEq)]
#[protocol(discriminant = "string")]
pub enum PlayerState {
  Stationary,
  Flying { velocity: (f32,f32,f32) },
  // Discriminators can be explicitly specified.
  #[protocol(discriminator("ArbitraryOverTheWireName"))]
  Jumping { height: f32 },
}

Misc

You can rename the variant for their serialisation.

#[derive(protocol::Protocol, Clone, Debug, PartialEq)]
#[protocol(discriminant = "string")]
pub enum Foo {
  Bar,
  #[protocol(name = "Biz")] // the Bing variant will be send/received as 'Biz'.
  Bing,
  Baz,
}

protocol's People

Contributors

anarelion avatar definitelynobody avatar dylan-dpc avatar dylanmckay avatar kern-- avatar tuxuser avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

protocol's Issues

Support for uuid seams broken.

I am having issues with getting the feature to use Uuid to work.

Cargo.toml:

[dependencies]
protocol-derive = "3.1.3"
protocol= { version = "3.1.3", features = ["uuid"] }
uuid="0.8.1"

Trying to compile something simple as

use uuid::Uuid;
use protocol_derive::Protocol;

#[derive(Protocol, Debug, Copy, Clone, PartialEq, Eq)]
pub struct Foo {
    pub id: Uuid,
}

And then I got a compiler error:

#[derive(Protocol, Debug, Copy, Clone, PartialEq, Eq)]
|        ^^^^^^^^ the trait `id::__impl_protocol___Parcel_FOR_Foo::protocol::HighLevel` is not implemented for `uuid::Uuid`

Am I missing something?

Null terminated strings

I am currently implementing a protocol that has 17 bytes or things that I don't care for now and then a 0 terminated string. It is datagram based, so the length of the packet is very clear from the start.

I am not the creator of the protocol, it has been around for 20 years or so.

Is there any way I can decode that string? What happens if there are bytes left to consume?

Merge the `Type` and `Packet` traits

Currently, there is the distinction between a Type that can be serialised, and a Packet that can be serialised. They both share almost the exact same definition.

It would be good to merge the two traits into one. This would allow connections to be build using primitives as packets, and also make everything more consistent.

Does not work with non_blocking communication.

I have tried to use this with a non_blocking socket and I get some weird error:

panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }'
protocol-3.1.3/src/wire/stream/transport/simple.rs:116:30

Is non blocking sockets supported?

Where clauses ignored when deriving Protocol for traits that have where clauses

Currently

#[derive(Protocol)]
enum Event<E: Event> {
   // ...
}

will work.

However

#[derive(Protocol)]
enum Event<E> where E: Event {
   // ...
}

does not. protocol-derive only looks at the constraints on the generic types, it currently fails to look at the where clause constraints and add them to the new impl Parcel.

Something like this needs to be done

diff --git a/protocol-derive/src/lib.rs b/protocol-derive/src/lib.rs
index 517c224..dfeda42 100644
--- a/protocol-derive/src/lib.rs
+++ b/protocol-derive/src/lib.rs
@@ -37,7 +37,7 @@ fn impl_parcel(ast: &syn::DeriveInput) -> proc_macro2::TokenStream {
 /// Returns `(generics, where_predicates)`
 fn build_generics(ast: &syn::DeriveInput) -> (Vec<proc_macro2::TokenStream>, Vec<proc_macro2::TokenStream>) {
     let mut where_predicates = Vec::new();
-    let generics: Vec<_> = ast.generics.params.iter().enumerate().map(|(i, p)| {
+    let generics: Vec<_> = ast.generics.type_params().enumerate().map(|(i, p)| {
         match p {
             syn::GenericParam::Type(t) => {
                 let (ident, bounds) = (&t.ident, &t.bounds);

NOTE: this if 1/3rd of a proper fix. A proper fix would also check generics.lifetimes() and generics.const_params and copy the constraints to the new impl. Currently the patch would ignore lifetime parameters and break all traits with lifetime type parameters.

Crate Abandoned?

Hello @dylanmckay,

do you plan on continuing work on this crate?
Seeing as there are several open issues and PRs, may I suggest adding me as a maintainer, or perhaps transferring ownership to my account @Feriixu?

Support for validation and size limiting

Right now if the raw bytes parse correctly, the next step in most applications is to validate (sanitize) the data. This being an additional step is a risk (programmer may forget about validation) and waste of resources (protocol has to parse all fields before sanitation can run). It may also result in unnecessary allocations - let's say our protocol accepts Vec<_> and someone sends us 8GB of data. protocol will download, parse and allocate 8GB Vec, which will be rejected by the next instruction like if vec.len() > 64 { return; }.

I imagine code could look like this:

#[derive(Protocol)]
pub struct User {
    pub name: String,
    pub pub_key: [u8; 64],
}

impl User {
    fn validate_name(name: String) -> bool { /* some code */ }
    fn validate_pub_key(pub_key: [u8; 64]) -> bool { /* some code */ }
}

User::parse( // or `User::validate_from_raw_bytes`
    &data,
    &protocol::Settings::default()
).unwrap();

Alternatively, protocol could generate trait, so we could have different ways of validating the same struct, and stateful validators:

#[derive(Protocol)]
pub struct User {
    pub name: String,
    pub pub_key: [u8; 64],
}

// protocol generates:
//
// trait UserValidate {
//     fn validate_name(&mut self, name: String) -> bool;
//     fn validate_pub_key(&mut self, pub_key: [u8; 64]) -> bool;
// }

struct UserBasicValidator;
impl UserValidate for UserBasicValidator {
    fn validate_name(&mut self, name: String) -> bool { /* some code */ }
    fn validate_pub_key(&mut self, pub_key: [u8; 64]) -> bool { /* some code */ }
}

User::parse(
    &data,
    &protocol::Settings::default(),
    UserBasicValidator::new(),
).unwrap();

Not possible to get discriminant of received packet?

I am interested in retrieving the discriminant of the packet that I have received. Though it seams as this is not possible. And I only wish to ask if I have understood this correctly?

If we use the example from the documentation. What would hope would be possible would be to do the following:

    loop {
        if let Some(response) = connection.receive_packet().unwrap() {
            println!("{:?}", response.discriminant());
            break;
        }
    }

But to me it seams as the only way one can retrieve discriminators is through the enum that defines the protocol?

#[macro_use] extern crate protocol_derive;
#[macro_use] extern crate protocol;

#[derive(Protocol, Clone, Debug, PartialEq)]
pub struct Handshake;

#[derive(Protocol, Clone, Debug, PartialEq)]
pub struct Hello {
    id: i64,
    data: Vec<u8>,
}

#[derive(Protocol, Clone, Debug, PartialEq)]
pub struct Goodbye {
    id: i64,
    reason: String,
}

#[protocol(discriminant = "integer")]
#[derive(Protocol, Clone, Debug, PartialEq)]
#[repr(u16)]
pub enum PacketKind {
    #[protocol(discriminator(0x00))]
    Handshake(Handshake),
    #[protocol(discriminator(0xaa))]
    Hello(Hello),
    #[protocol(discriminator(0xaf))]
    Goodbye(Goodbye),
}

fn main() {
println!("Disc: {}", Packet::Handshake(Handshake)).discriminant());
}

In order to get the discriminant of message do I have to write my own function that checks the message type and return the appropriate discriminator by constructing a temporary object and extract it from the temporary object?

use protocol_derive;
use protocol::{ByteOrder, Enum, Parcel, Settings};

#[derive(Protocol, Clone, Debug, PartialEq)]
pub struct Handshake;

#[derive(Protocol, Clone, Debug, PartialEq)]
pub struct Hello {
    id: i64,
    data: Vec<u8>,
}

#[derive(Protocol, Clone, Debug, PartialEq)]
pub struct Goodbye {
    id: i64,
    reason: String,
}

#[protocol(discriminant = "integer")]
#[derive(Protocol, Clone, Debug, PartialEq)]
#[repr(u16)]
pub enum PacketKind {
    #[protocol(discriminator(0x00))]
    Handshake(Handshake),
    #[protocol(discriminator(0xaa))]
    Hello(Hello),
    #[protocol(discriminator(0xaf))]
    Goodbye(Goodbye),
}

impl PacketKind {
pub fn get_discriminator(&self) -> u16 {
match self {
    PacketKind::Handshake(_) => {
        PacketKind::Handshake(Handshake).discriminator()
    },
    PacketKind::Hello(_) => {
       PacketKind::Hello(Hello {id: Default::default(), data: Default::default() }).discriminant()
    },
    PacketKind::GoodBye(_) => {
        PacketKind::Goodbye(GoodBye {id: Default::default(), reason: "".to_string() }).discriminant()
    },
}

It seam really weird that the library user would have to go through all this trouble to get the discriminant of the received package or am I missing something?

remove panic on unknown discriminator

This panic line caused one of our production systems to crash when trying to decode messages on a noisy channel.

_ => panic!("unknown discriminator"), // FIXME: this should not be a panic

Now, I'm happy to fix this and submit a pull request but how? Because this is part of the derived parcel implementation, I'm not sure what the generated code ends up looking like or what can be returned here to replace the panic

Variable protocol depending on context: Custom Settings or Hints.

The protocol I am implementing has certain quirks. It is a opcode u16 based protocol, so first 2 bytes define the rest, but:

Some packets can be embebbed into other packets and depending on that, a CRC is added at the end or not.

Does protocol has a easy way to map this? Is there a way that I can provide custom settings or hints ?

Doesn't work at latest nightly rust toolchain

error[E0271]: type mismatch resolving `<i64 as std::convert::TryFrom<i8>>::Error == std::num::TryFromIntError`
   --> ...\protocol-0.3.1\src\primitives\numerics.rs:102:6
    |
102 | impl Integer for i64 { }
    |      ^^^^^^^ expected enum `std::convert::Infallible`, found struct `std::num::TryFromIntError`
    |
    = note: expected type `std::convert::Infallible`
               found type `std::num::TryFromIntError`

Support transmission of structs as name:value field pairs

In 19efbfa, a new #[protocol(name = "foo")] attribute was introduced for enum variants.

We should support transmission of structures as field_name:field_value pairs so that fields may be added.

When this is done, we should also implement support for the #[protocol(name = "afd")] for each field.

Fixed length

Hello , is it possible to set a fixed length for a string field ? I would need something like this :

#[derive(Protocol, Debug, Clone, PartialEq, Eq)]
pub struct DatFile {
    #[protocol(fixed_lenght(8))]
    game_version: String,
}

Support array sizes over 32

Hey,

I'm currently working on a Macho parser and couldn't figure out how to declare sized char arrays (char foo[16]). I tried [u8; 16], with no luck.

Is that possible right now at all?

Thanks!

Rust 2018

Hello.

I wanted to write a small tool to calculate something distributed on a master with slaves and learn more about Rust on the way. Having a simple transport layer with the help of protocol seemed like a good way. I'm using rustc 1.31.0 (abe02cefd 2018-12-04) and am currently getting this error:

error[E0107]: wrong number of type arguments: expected 0, found 2                                                                                                                                                                       
 --> src/network.rs:4:10                                                                                                                                                                                                                
  |                                                                                                                                                                                                                                     
4 | #[derive(Protocol, Debug, PartialEq)]                                                                                                                                                                                               
  |          ^^^^^^^^ 2 unexpected type arguments                                                                                                                                                                                       

I'm trying to use this on:

#[derive(Protocol, Debug, PartialEq)]
pub enum Message {
    RequestRegistration,
    Register(Registration),
    Job(Job),
    Result(Job, Result),
    RequestStatus,
    Status(Status),
}

Sadly the error is not very helpful in explaining what exactly I'm doing wrong but I highly suggest it has to do with Rust 2018, because the examples still use extern crate and #[macro_use]. Also there's no edition = "2018" in the Cargo.toml.

Any help on this is very appreciated. I don't think there's any other crate that can do what procotol suggests it can do this easily.

Support optionally-enabled middleware

We could add impl Middleware for Option<M: Middleware>.

Then piplines could be done line so

struct Pipeline {
     encryption: Option<Encryption>,
     compression: Option<Compression>,
}

fn handle(packet: _, pipeline: &mut Pipeline) {
    match packet {
        LoginSuccess => {
            pipeline.encryption = Some(Encryption::new(..));
        },
    }
}

This would be required for protocols such as minecraft, which optionally enables encryption at login.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.