google / pdl Goto Github PK

License: Apache License 2.0

Python 11.25% C++ 0.28% Rust 87.52% Shell 0.95%

rust parsing

pdl's Introduction

Packet Description Language (PDL)

PDL is a domain specific language for writing the definition of binary protocol packets. Parsing and validating packets from raw bytes is tedious and error prone in any language. PDL generates memory safe and tailored backends for multiple target languages:

- Rust
- C++
- Python

Historically PDL was developed as part of the Android Bluetooth stack (bluetooth_packetgen) as a way to generate the parser and serializer for Bluetooth packets, and reduce the number of memory safety issues that come with manipulating and validating raw data.

How to use PDL

Write the protocol definition
cargo run my-protocol.pdl --output-format rust > my-protocol.rs

Language specific instructions are provided for all supported backends:

Supported Features

Full reference documentation

Scalar values
Enumerators
Arrays
Optional fields
Nested packets
Conditional packet derivation
Custom field definitions

Similar projects

pdl's People

Contributors

Stargazers

Watchers

Forkers

hchataing sarvex mgeisler cemberk isabella232 uael colecf studgeek

pdl's Issues

Whitespace is optional between keywords

Surprisingly, this PDL file is valid:

little_endian_packetsstructx{foo:8}

😄

We should probably require at least one space between our tokens.

CRLF files are not currently supported

If file is stored using CRLF convention the parser fails with very cryptic message.

Feature request: Packet API that supports writing to a mutable Vec

Currently if I want to prepend any bytes to the Vec that is generated by to_vec, I need to create a new Vec, write my desired bytes, then copy the Vec contents generated by to_vec.

However, it would be nice if a Packet had a function such as append(&mut Vec).
This would eliminate the need to copy all the packet bytes, and write directly into the Vec that I have already initialized to my liking.

Make generated rust no_std compatible

Feature request: derive `PartialEq` for `enum Error`

As part of my implementation, I define a custom Error, one variant which wraps an Error enum that is generated by PDL.

In some of my tests, I check that I get the expected error types, which is simplified if I derive PartialEq for my Error enum.

Problem is, I have to implement PartialEq for the PDL-generated Error enum if I want one of my error variants to wrap it.

Is it feasible to derive PartialEq for Error enums?

Implement support for structure inheritance in the rust generator

The following grammar is not supported:

struct Parent {
    x: 8,
    _payload_,
}

struct Child : Parent {
    y: 8
}

endianess should include host-order

PROBLEM

netlink.h and mac80211_hwsim.h packets are defined in host order, however only "little-endian" and "big-endian" are allowed.

These definitions are used for "simulated" networking between kernel and user space and also by VMs from guest to host to pass Wi-Fi packets to/from the linux mac80211_hwsim driver.

See https://wireless.wiki.kernel.org/en/users/drivers/mac80211_hwsim

RECOMMENDATION

Add another type: "host-endian"

Unexpected payload size for non byte aligned payload

The following PDL file crashes backends::rust::generate, even though it passes through the analyzer:

little_endian_packets

struct Foo {
  ts_sec: 20,
  ts_usec: 32,
  _size_(_payload_): 32,
  _payload_,
}

Stack trace:

thread '<unnamed>' panicked at 'Unexpected payload size for non byte aligned payload', /home/mgeisler/src/pdl/src/backends/rust/parser.rs:526:17
stack backtrace:
   0: begin_panic_handler
             at /rustc/b2b34bd83192c3d16c88655158f7d8d612513e88/library/std/src/panicking.rs:593:5
   1: panic_fmt
             at /rustc/b2b34bd83192c3d16c88655158f7d8d612513e88/library/core/src/panicking.rs:67:14
   2: pdl_compiler::backends::rust::parser::FieldParser::add_payload_field
             at ./src/backends/rust/parser.rs:526:17
   3: pdl_compiler::backends::rust::parser::FieldParser::add
             at ./src/backends/rust/parser.rs:77:17
   4: pdl_compiler::backends::rust::generate_data_struct
             at ./src/backends/rust.rs:236:9
   5: pdl_compiler::backends::rust::generate_struct_decl
             at ./src/backends/rust.rs:696:38
   6: pdl_compiler::backends::rust::generate_decl
             at ./src/backends/rust.rs:979:13
   7: pdl_compiler::backends::rust::generate::{{closure}}
             at ./src/backends/rust.rs:998:53

Array element size (20) is not a multiple of 8

The following PDL file crashes backends::rust::generate despite passing through the analyzer:

little_endian_packets

struct PcapHeader {
  _fixed_ = 0xa1b2cd4: 32,
}

struct PcapRecord {
  ts_sec: 20,
}

packet PcapFile {
  header: PcapHeader,
  records: PcapRecord[],
}

Stack trace:

thread '<unnamed>' panicked at 'assertion failed: `(left == right)`
  left: `4`,
 right: `0`: Array element size (20) is not a multiple of 8', /home/mgeisler/src/pdl/src/backends/rust/parser.rs:289:17
stack backtrace:
   0: begin_panic_handler
             at /rustc/b2b34bd83192c3d16c88655158f7d8d612513e88/library/std/src/panicking.rs:593:5
   1: panic_fmt
             at /rustc/b2b34bd83192c3d16c88655158f7d8d612513e88/library/core/src/panicking.rs:67:14
   2: assert_failed_inner
   3: core::panicking::assert_failed
             at /rustc/b2b34bd83192c3d16c88655158f7d8d612513e88/library/core/src/panicking.rs:228:5
   4: pdl_compiler::backends::rust::parser::FieldParser::add_array_field
             at ./src/backends/rust/parser.rs:289:17
   5: pdl_compiler::backends::rust::parser::FieldParser::add
             at ./src/backends/rust/parser.rs:67:71
   6: pdl_compiler::backends::rust::generate_data_struct
             at ./src/backends/rust.rs:236:9
   7: pdl_compiler::backends::rust::generate_packet_decl
             at ./src/backends/rust.rs:579:48
   8: pdl_compiler::backends::rust::generate_decl
             at ./src/backends/rust.rs:972:45
   9: pdl_compiler::backends::rust::generate::{{closure}}
             at ./src/backends/rust.rs:998:53

Make generated rust no_panic compatible

Have a way to represent the size of two equal data fields

Two options that we talked about are:

packet TwoEqualFields {
  size_of_fields : 8,
  field_one : 8[size_of_fields],
  field_two : 8[size_of_fields],
}

packet TwoEqualFields {
  _elementsize_(data) : 8,
  data: 8[2][],  // Two variable-size byte-arrays that have the same size.
}

The benefit of elementsize is that we will need something like that for GATT.
The downside is that data and mask are not the same, so it would be nicer to be able to have meaningful names.

Two other options are:

packet TwoEqualFields {
  _size_(field_one) : 8,
  field_one : 8[],
  field_two : 8[_size_(field_one)],
}

packet TwoEqualFields {
  _size_(field_one) {, && ==} _size_(field_two) : 8,
  field_one : 8[],
  field_two : 8[],
}

Implement TryInto for Owned values and not only their reference

To allow doing

 MyChild { a: 42 }.try_into()

instead of

(&MyChild { a: 42 }).try_into()

Include definitions from other files

Some of my structures are re-used in multiple places. Also I don't want to put dozens of structures into one giant file. I would prefer to have one file per packet. However currently I don't see a way to refer to packet defined in another file. Of cause the parser fails with "undeclared typedef identifier ...".

Implement support for optional fields

Optional fields are present in a number of binary packet specification; where a bit flag is used to determine the presence / absence of a field. This task aims to enable support for the following king of specification:

packet {
    a : 1,
    opt_a : A if a = 1,
}

The limitations are to be discussed:

limited to boolean flags or not ?
limited to typedef / scalar fields or not ? e.g. does it make sens to allow this flag for size fields.

Use relative file path in `pdl` proc macro attribute

The API Span::source_file is currently nightly (https://doc.rust-lang.org/stable/proc_macro/struct.Span.html#method.source_file), and there is no alternate way to get the path to the file a proc macro attribute runs on.
This means the pdl proc_macro attribute cannot use a relative file path similarly to include_str!.
This should be updated when the API becomes stable.

Implement support for payload size modifiers in the rust generator

struct Test {
    _payload_: [+4],
}

Add support for variant constraint values to PDL packet declarations

Example use case taken from the UWB's UCI specification:

enum MessageType: 3 {
    DATA = 0x00,
    COMMAND = 0x01,
    RESPONSE = 0x02,
    NOTIFICATION = 0x03,
}

packet UciPacket {
    message_type: MessageType,
    _body_
}

packet UciControlPacket: UciPacket (message_type = COMMAND | RESPONSE | NOTIFICATION) {
    opcode: 8,
    _size_(_payload_): 8,
    _payload_,
}

packet UciDataPacket: UciPacket (message_type = DATA) {
    _size_(_payload_): 16,
    _payload_,
}

In this example UciControlPacket covers multiple message types. It is not possible to describe this constraint using the exisiting syntax.

Add canonical test cases for invalid packet inputs

The current test vectors only test for valid inputs, and are lacking tests to verify that the parsers are correctly rejecting invalid packets without crashing.

Implement snapshot tests and canonical tests for github actions

Ensure that python and cxx generator tests are being run from github actions

Create codegen library for use in build.rs

As reference https://crates.io/crates/protobuf-codegen
Create a library that can facilitate the invocation of pdlc from build.rs files.

Potential interface:

pdlc::Codegen::new()
   .input("test.pdl")
   .output("test.rs")
   .output_format(Codegen::Format::Rust)
   .run();

Support Boolean type

I couldn't find a way to use bool. Interim I am trying to use:

enum Bool: 1 {
    false = 0,
    true = 1
}

Feature request: infallible "from" for custom fields with bits that are a multiple of 8

If a custom field takes up, say, 48 bits, and each bit can be arbitrarily anything, then it should be able to infallibly be converted from a [u8; 6].

But the current generated code performs a fallible try_from(u64), which would require wrangling a series of bytes into a [u8; 8], and then converted to a u64.