gilnaa / memoffset Goto Github PK
View Code? Open in Web Editor NEWoffsetof for Rust
License: MIT License
offsetof for Rust
License: MIT License
offset_of! (and probably the other macros in this crate) consumes too much stack space in dev build, especially when the type is big. It's probably because it's declaring a (uninitialized) variable of the type. It's optimized out in the release build.
It affects my workflow in developing an OS in Rust. Its dev build won't boot because offset_of! of a big struct (2KiB) used up the limited amount of kernel stack (4KiB), incurring stack overflow.
[This is perhaps more of a documentation request - it's information I might have expected to find in the README]
I understand that Rust generally does not make any guarantees regarding the layout of structs. The book states:
Type layout can be changed with each compilation. Instead of trying to document exactly what is done, we only document what is guaranteed today.
If you apply offset_of!
to a struct
with the default representation, do you get garbage, or do you get whatever the offset is under the current compiler and current compilation pass?
Current implementation creates a dummy object. It's possible to get alignment with just pointer manipulation.
let base = std::mem::align_of::<Foo>(); // First valid non-NULL pointer address
let ptr = base as *const Foo;
assert_eq!(0, (&(*ptr).field) as *const _ as usize - base);
I think &(*ptr)
is valid and does not actually deference the object, because such construct is also allowed for non-movable values (where *obj
causes an error, but &*obj
is fine).
I will report https://www.cvedetails.com/cve/CVE-2019-15553/ to the Rust security advisory database. So it becomes possible to automatically detect this CVE in Rust CIs. I create this issue to track it.
Playground containing an extract of the code from the current master (d3f5f23)
#[repr(C)]
struct Foo {
a: core::sync::atomic::AtomicUsize,
}
assert_eq!([0; offset_of!(Foo, a)].len(), 0);
Building this fails with the following error:
error[E0492]: cannot borrow a constant which may contain interior mutability, create a static instead
--> src/offset_of:112:66
|
112 | let base_ref = ::core::mem::transmute::<_, &$parent>(&uninit);
| ^^^^^^^
...
NNN | assert_eq!([0; offset_of!(Foo, a)].len(), 0);
| ------------------ in this macro invocation
Note that there is no error if Foo::a
is usize
instead.
memoffset does not compile with actual nightly:
error[E0635]: unknown feature
const_transmute
It's very likely that https://docs.rs/autocfg/1.0.0/autocfg/ becomes the crate for rustc feature detection:
It might make sense to replace rustc_version
with autocfg
Hi! Thank you for this crate. I was going to update it in our dependency tree and wanted to check what the changes were. But I'm not able to find a changelog. I can of course not be lazy and do a git diff. But having a human readable changelog is really nice. A changelog can outline caveats to not miss and other things that might not be obvious from reading just the code diff. So this is just a friendly request to have one added :)
memoffset 0.5.1 and 0.5.3 cause clippy to warn in the consumer's crate. This should either be fixed or suppressed within memoffset.
warning: All the struct fields are matched to a wildcard pattern, consider using `..`.
--> bfffs-fio/src/lib.rs:95:17
|
95 | offset_of!(BfffsOptions, pool_name),
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= note: `#[warn(clippy::unneeded_field_pattern)]` on by default
= help: Try with `BfffsOptions { .. }` instead
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#unneeded_field_pattern
= note: this warning originates in a macro outside of the current crate (in Nightly builds, run with -Z external-macro-backtrace for more info)
There's a cfg(memoffset_constant_expression)
in there that never applies.
Travis stopped working for a few projects of mine, it can probably stop working any day here as well. (Looks like projects are granted a specific number of "credits" overall and once those are out, it's over.)
We should probably switch to GHA. I will prepare a PR.
offset_of
is UB because we are creating a reference to a field that has not been initialized yet. (Miri does not currently check references transitively, so tests still pass in Miri.) There is currently no way to avoid this UB when implementing offset_of!
. progress is blocked on rust-lang/rfcs#2582.
We are also "more UB" for older versions of rustc, because we are dereferencing a dangling pointer. We only do it to compute a field address, so no memory access happens, but the dereference is enough to cause UB.
See #9 for other ways in which there used to be "more UB".
Version 0.4 contains the union-transmute-based version of span_of
corresponding to the offset_of
in the yanked 0.3. Would it be a good idea to yank 0.4 as well?
Previously on memoffset:
Constant evaluation mostly stable with the exception of the inability to take address of a Cell in const
which still requires nightly and #![feature(const_refs_to_cell)]
.
span_of
creates a reference to an uninitialized field for size_of_val
.
We should switch to a different implementation that works with raw pointers. It doesn't even have to support unsized types (we just need the type inference), so we could use
fn size_of_pointee<T>(_ptr: *const T) -> T {
mem::size_of::<T>()
}
The primary macro implementations in memoffset 0.5 use other secondary macros, but don't scope them with $crate (or something like that, I don't actually know what the proper syntax is).
This causes a problem at the use site: if only the main macro is imported, the macro invocation expands to code that won't compile.
use memoffset::{offset_of, span_of};
struct Hmm {
mmm: u32,
}
fn main() {
println!("{}", offset_of!(Hmm, mmm));
println!("{}", span_of!(Hmm, mmm));
}
error: cannot find macro `field_check!` in this scope
--> src/main.rs:8:20
|
8 | println!("{}", offset_of!(Hmm, mmm));
| ^^^^^^^^^^^^^^^^^^^^
|
= note: this error originates in a macro outside of the current crate (in Nightly builds, run with -Z external-macro-backtrace for more info)
error: cannot find macro `let_base_ptr!` in this scope
--> src/main.rs:8:20
|
8 | println!("{}", offset_of!(Hmm, mmm));
| ^^^^^^^^^^^^^^^^^^^^
|
= note: this error originates in a macro outside of the current crate (in Nightly builds, run with -Z external-macro-backtrace for more info)
error: cannot find macro `let_base_ptr!` in this scope
--> src/main.rs:9:20
|
9 | println!("{}", span_of!(Hmm, mmm));
| ^^^^^^^^^^^^^^^^^^
|
= note: this error originates in a macro outside of the current crate (in Nightly builds, run with -Z external-macro-backtrace for more info)
error: cannot find macro `field_check!` in this scope
--> src/main.rs:9:20
|
9 | println!("{}", span_of!(Hmm, mmm));
| ^^^^^^^^^^^^^^^^^^
|
= note: this error originates in a macro outside of the current crate (in Nightly builds, run with -Z external-macro-backtrace for more info)
error[E0425]: cannot find value `base_ptr` in this scope
--> src/main.rs:8:20
|
8 | println!("{}", offset_of!(Hmm, mmm));
| ^^^^^^^^^^^^^^^^^^^^ not found in this scope
|
= note: this error originates in a macro outside of the current crate (in Nightly builds, run with -Z external-macro-backtrace for more info)
error[E0425]: cannot find value `base_ptr` in this scope
--> src/main.rs:8:20
|
8 | println!("{}", offset_of!(Hmm, mmm));
| ^^^^^^^^^^^^^^^^^^^^ not found in this scope
|
= note: this error originates in a macro outside of the current crate (in Nightly builds, run with -Z external-macro-backtrace for more info)
error[E0425]: cannot find value `root` in this scope
--> src/main.rs:9:20
|
9 | println!("{}", span_of!(Hmm, mmm));
| ^^^^^^^^^^^^^^^^^^ not found in this scope
|
= note: this error originates in a macro outside of the current crate (in Nightly builds, run with -Z external-macro-backtrace for more info)
error[E0425]: cannot find value `root` in this scope
--> src/main.rs:9:20
|
9 | println!("{}", span_of!(Hmm, mmm));
| ^^^^^^^^^^^^^^^^^^ not found in this scope
|
= note: this error originates in a macro outside of the current crate (in Nightly builds, run with -Z external-macro-backtrace for more info)
error: aborting due to 8 previous errors
It would be cool to get a release out which includes #50 , now that 1.51 is released.
This doesn't work with memoffset:
struct Foo {
bar: f32,
baz: [f32],
}
offset_of!(Foo, bar);
Even though this (unsound) code works:
struct Foo {
bar: f32,
baz: [f32],
}
let foo_ptr: &Foo = unsafe { mem::zeroed() };
let foo_offs = (&foo_ptr.bar) as *const _ as usize;
I ran into this problem when trying to port glium to memoffset in glium/glium#1782. The gpgpu example has relied on offset of calculations supporting unsized structs. I could work around the issue by changing the gpgpu example, but that isn't really perfect.
I know it's hard/impossible right now, but are you tracking the ongoing const_fn
changes to ensure that offset_of!
can be used in constants at some point? It's usually very useful in tandem with size_of()
, which is already const fn
.
The feature const_fn
is no longer supported on Rust nightly, hence memoffset 0.6.3 no longer compiles when using the unstable_const
feature.
memoffset does not work with unions. Would it be possible to add this feature?
For example, this code:
use memoffset::offset_of;
pub union Foo {
foo32: i32,
foo64: i64
}
#[test]
fn alignment() {
assert_eq!(offset_of!(Foo, foo32), offset_of!(Foo, foo64));
}
Produces this result:
error: `..` cannot be used in union patterns
--> test/memoffset.rs:10:16
|
10 | assert_eq!(offset_of!(Foo, foo32), offset_of!(Foo, foo64));
| ^^^^^^^^^^^^^^^^^^^^^^
|
= note: this error originates in the macro `_memoffset__field_check` (in Nightly builds, run with -Z macro-backtrace for more info)
error: `..` cannot be used in union patterns
--> test/memoffset.rs:10:40
|
10 | assert_eq!(offset_of!(Foo, foo32), offset_of!(Foo, foo64));
| ^^^^^^^^^^^^^^^^^^^^^^
|
= note: this error originates in the macro `_memoffset__field_check` (in Nightly builds, run with -Z macro-backtrace for more info)
Some crates.io releases do not have corresponding tags in the repository: 0.5.1, 0.2.1, 0.2.0.
We should have compile-fail tests (as doctests, I heard that's possible) to make sure we properly check for deref coercions, and whatever else might be relevant.
With the implementation in:
macro_rules! offset_of {
($parent:ty, $($field:tt)+) => (unsafe {
let x: &'static $parent = $crate::Transmuter::<$parent> { int: 0 }.ptr;
$crate::Transmuter { ptr: &x.$($field)+ }.int
});
}
you are constructing a &'static T
at address 0
. This is undefined behavior because Rust may assume that you have a valid &'static T
but you do not.
Moreover, in the old pre-1.33.0 implementation you have:
macro_rules! offset_of {
($father:ty, $($field:tt)+) => ({
#[allow(unused_unsafe)]
let root: $father = unsafe { $crate::mem::uninitialized() };
let base = &root as *const _ as usize;
// Future error: borrow of packed field requires unsafe function or block (error E0133)
#[allow(unused_unsafe)]
let member = unsafe { &root.$($field)* as *const _ as usize };
$crate::mem::forget(root);
member - base
});
}
Note that when you say &root
you have already triggered undefined behavior because again, you assert with &root
that you have a valid reference but you do not in fact.
See rust-lang/rfcs#2582 and rust-lang/unsafe-code-guidelines#77 for a discussion.
As for your comment about // Future error
, do note that unsafe { ... }
here does not make what you are doing any less undefined behavior. Instead, while you might not get an error from the compiler, you might get miscompilation should LLVM or rustc's behavior change.
cc @RalfJung
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.