ellacrity / abio Goto Github PK
View Code? Open in Web Editor NEWLow-level, endian-aware routines for zero-copy decoding and encoding
License: Apache License 2.0
Low-level, endian-aware routines for zero-copy decoding and encoding
License: Apache License 2.0
This project will soon be made public and published to crates.io when v0.3.0 is stabilized.
I have decided that abio
will be using "trunk-based development". Within the context of this library, that means that there will be no long-lived feature branches. The foundation of the library has been implemented and I am happy with the shape of the API.
With that said, the most important thing to take away from this is that this library will use a trunk-based approach to development such that any pushes to the main
branch must pass all existing tests and all workflows must succeed. Broken code should never be pushed to main
.
This crate needs additional unit testing and, more importantly, integration testing.
The proposed test suite will involve miri
to help detect UB, and the addition of a few more dev-dependencies
to make it easier to write high-quality, meaningful tests.
In addition to this, I would like to add a basic benchmarking suite to start, and then it can be expanded upon from there. The most important functions to benchmark are those that involve reading data into slices vs. arrays (Bytes
/ Chunk<N>
types).
Ideally, this will make it into v0.3.0 to give end-users a better picture of what the crate's capabilities are. There are not many crates in this particular area (zero copy direct reading/writing from byte slices aka "safe transmute"), so it is crucial that abio
is able to compete with these crates.
Currently, the traits exposed via the public API are incomplete. Traits are being rearranged, with trait bounds being relaxed in some cases, while in others they are more restrictive.
This crate is intended to provide a more ergonomic interface and API for working with raw byte buffers. It is absolutely imperative that the types and trait implementations in [abio
] are sound. At the cost of a small runtime penalty, bounds checks, alignment checks on pointers (that cannot be optimized away due to not being evaluated within const
context), etc., This crate makes working in extremely hazardous environments simpler, easier and more intuitive.
This marks an absolute major release for this project. The entire code base has been refactored, traits have been modified to work together and produce an API that is both easy to use and leverages zero-cost abstractions as much as possible to push as much evaluation as possible to oonst
contexts.
For additional information about const
contexts and their relevance, please refer to reference section on const evaluation
This project is an experiment concerning "Compile Time Function Evaluation" (CTFE) and how it may be used to create components such as lexers, parsers, memory mappers, and anything else that relies on the use of zero-copy, endian-aware operations against raw bytes/memory.
I want to push CTFE to its limits, because I believe it can be used to improve our collective tooling as developers and it can do so safely (by evaluating as much as possible at compile time) with minimal runtime cost.
Unfortunately, Rust simply lacks the required features to realize this project. As powerful as it is, many of the needed features are still unstable and may change. The initial release of "const generics" was a step in the right direction, but things like "generic const expressions" are not even close to being stable.
Procedural macros are absolutely one solution to this problem. However, I am not willing to take on that amount of technical debt when it is not necessary. Rust meta-programming is quite complex and is not at all something I am interested in delving into for the sake of this project.
Having a lot of procedural macros may generate a significant amount of code, which can cause bloating of end-user binary/library sizes, very long compilation times (relatively speaking) and they are frankly just not easy to understand for most people.
This poses then yet another problem: if this library relies on complex procedural macros, it will be inherently more difficult to find contributors to help out. As powerful as they are, procedural macros are arguably arcane and esoteric in nature. If I had to guess, most Rust developers are probably not experts with procedural macros.
For the reasons stated above, along with several others, I have decided to begin porting this project from Rust to Zig. A new repository will be created, separate from this one, but some things will be copied over verbatim.
This project is still under heavy development. The new repository will contain any information needed to contribute, if you are interested.
[Chunk](https://docs.rs/abio/0.3.0/abio/struct.Chunk.html)
A newtype wrapper for fixed-size byte arrays (generic over the constant N parameter).
[Slice](https://docs.rs/abio/0.3.0/abio/struct.Slice.html)
A newtype wrapper for byte slices.
Note: The Slice type will be renamed to Bytes in the upcoming 0.4.0 release.
The Bytes
(formerly Slice
) and Chunk<N>
types serve as core primitives for handling byte slices and arrays. Their importance is emphasized by their extensive use throughout the codebase. Moreover, there are plans to further extend their APIs and functionalities. In other words, these types are not going anywhere anytime soon.
Introducing these types into the public API demands caution. Their primary purpose is to encapsulate unsafe
code, offering safe interfaces in return. While "Deref polymorphism" might be tempting due to its convenience, it appears to be actively harmful in the context of this library. Its inherently risky nature presents numerous pitfalls. As a result, retaining the status quo for these types feels inappropriate.
The first solution that came to mind was to simply remove these types from the public API entirely. This change would eliminate the potential misuse and these types would remain, but serve as mere implementation details.
Instead, I think that removing the current Deref
implementations is both less invasive and more useful. As such, the code will be refactored to function without relying on "Deref polymorphism." This approach requires minimal adjustments and negates the necessity of removing types from the public API.
Bytes
/ Chunk<N>
By keeping the types public while reducing their API scope, we can safely expose these types and allow to benefit from pending niche optimizations. Moreover, it paves the way for enhancing the utility and applicability of these types. My motivation for originating this library stemmed from my dissatisfaction with prevailing solutions; I yearned for a more flexible alternative.
It would be great to introduce some additional utility types for FFI, which is a major use-case for this library. Introducing variants such as Utf8Str
for working with Rust's &str
types, or NullStr
to handle C-style null-terminated strings, would be welcome additions.
One of the core "values" of this library is that it provides safety guarantees to users by imposing restrictions. This is how Rust works as a whole, so it makes sense to me to use what the language offers.
Overall, this should result in safer code and allow more aggressive optimizations for these types. The more restricted they are, the more we can play fast and loose elsewhere.
The decision was made to make the error
module largely private, or pub(crate)
. Very little of it is actually exposed. As of this moment, the public API contains Error
and a Result
type, but there is no way to use the constructor functions local to the crate to instantiate abio::Error
types.
This library is currently still quite unstable. With the release of 0.4.0, the API will be significantly more stable and the project will be heading in the right direction.
This is largely an enhancement and not a priority at the moment. If someone feels like picking up this issue, please feel free. I could use the help and this could help make the library more performant.
The main issue that I see right now is that the library still lacks a proper benchmarking suite. This will be implemented as part of version 0.4.0, so once the 0.4.0 milestone is reached, this issue can be addressed.
There is, as of right now, an insufficient amount of testing to definitively say that the following implementations are sound:
unsafe impl<T: Abi> Abi for *const T {}
unsafe impl<T: Abi> Abi for *mut T {}
It may be prudent to impose additional restrictions via trait bounds for these blanket implementations. However, I think it makes a bit more sense right now, with this library being in its absolute infancy, to allow this until it can be proven to be sound or unsound.
This crate is meant to be flexible, yet safe. Flexibility and ergonomics are a very high priority for abio
, so I will leave this issue open until the question is resolved.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.