gnzlbg / ctest Goto Github PK

View Code? Open in Web Editor NEW

125.0 4.0 28.0 993 KB

Automatic testing of FFI bindings for Rust

Home Page: https://docs.rs/ctest

License: Apache License 2.0

Rust 92.33% C 5.95% Dockerfile 0.21% Shell 1.50%

ctest's Introduction

ctest

Documentation

Automated testing of FFI bindings in Rust. This repository is intended to validate the *-sys crates that can be found on crates.io to ensure that the APIs in Rust match the APIs defined in C.

Example

Unfortunately the usage today is a little wonky, but to use this library, first, create a new Cargo project in your repo:

$ cargo new --bin systest

Then, edit systest/Cargo.toml to add these dependencies:

[package]
# ...
build = "build.rs"

[dependencies]
mylib-sys = { path = "../mylib-sys" }
libc = "0.2"

[build-dependencies]
ctest = "0.2"

Next, add a build script to systest/build.rs:

extern crate ctest;

fn main() {
    let mut cfg = ctest::TestGenerator::new();

    // Include the header files where the C APIs are defined
    cfg.header("foo.h")
       .header("bar.h");

    // Include the directory where the header files are defined
    cfg.include("path/to/include");

    // Generate the tests, passing the path to the `*-sys` library as well as
    // the module to generate.
    cfg.generate("../mylib-sys/lib.rs", "all.rs");
}

Next, add this to src/main.rs

#![allow(bad_style)]

extern crate mylib_sys;
extern crate libc;

use libc::*;
use mylib_sys::*;

include!(concat!(env!("OUT_DIR"), "/all.rs"));

And you're good to go! To run the tests execute cargo run in the systest directory, and everything should be kicked into action!

How it works

This library will parse the *-sys crate to learn about all extern fn definitions within. It will then generate a test suite to ensure that all function function signatures, constant values, struct layout/alignment, type size/alignment, etc, all match their C equivalent.

The generated tests come in two forms. One is a Rust file which contains the main function (hence the include! above), and another is a C file which is compiled as part of the build script. The C file is what includes all headers and returns information about the C side of things (which is validated in Rust).

A large amount of configuration can be applied to how the C file is generated, you can browse the documentation.

Projects using ctest

License

This project is licensed under either of

Apache License, Version 2.0, (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in ctest by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

ctest's People

Contributors

Stargazers

Watchers

ctest's Issues

Don't use #[link_name] for the C symbol

In the libc crate, NetBSD, OS X and Windows all use #[link_name] to sometimes use alternative linker symbols for functions like stat.

As far as I can tell, on OS X and NetBSD the system header files do a similar technique: The C symbol is distinct from the linker symbol. In the case of NetBSD for example, there is no C symbol named __stat50, the C source code keeps using the original stat C symbol. Only for the assembly output the linker symbol __stat50 is used (coming from an asm directive in the C header file).

On Windows however, C source code needs to use the C symbol _stat directly (which is also used for the linker symbol).

Currently, ctest uses the linker symbol as the C symbol for the generated C code. This is needed for Windows, since there the C symbol is actually the symbol used in #[link_name]. But it breaks on NetBSD, as there the #[link_name] linker symbols are undefined in the C compiler.

I'm not sure how to fix this. It seems that there already is an rule excluding symbols containing $ for OS X (like stat$INODE64). We could add a rule to exclude symbols starting with __ to work around this issue for NetBSD, but it would just be an additional hack.

static function pointers are broken

Examples:

extern "C" {
#[no_mangle]
pub static mut bar:
extern fn (a: *mut u8, b: *const u16);
}

errors on MacOS with

error: expected function body after function declarator
cargo:warning=            extern fn(u8, u16) -> ()* __test_static_bar(void) {

The problem seems to be that the following invalid C code is generated for these:

extern fn(u8, u16) -> ()* __test_static_bar(void) {
    return &bar;
}

That is, it appears that the Rust function type extern fn(u8, u16) -> () is not being translated to the C function type;

Panic in syntex-syntax on ctest 0.1.5

Not sure if this is an issue with syntex-syntax or with how ctest is invoking it, but I'm seeing panics: https://circleci.com/gh/sfackler/rust-openssl/1437. Seems like it's something in https://github.com/sfackler/rust-openssl/blob/master/openssl-sys/src/ossl10x.rs but I haven't narrowed it down beyond that.

Test C Macros with custom code

C macros and inline functions generally have to be reimplemented in Rust. That's error-prone and leads to bugs like rust-lang/libc#1210 and rust-lang/libc#1190 . Could ctest support testing macro reimplementations with custom code? The general idea would be to compare the output of the C macro against the output of the Rust function, with the same inputs. Implementation would probably require wrapping each C macro with a wrapper that's callable from Rust.

Use of macros causes panic in syntex_syntax

Using a macro can cause unreachable code in syntex_syntax to get hit. For example if you change this test line to:

pub const T1S: &'static str = stringify!(foo);

then the test panics.

Use of uninitialized warns on 1.36.0

warning: the type `unsafe extern "C" fn() -> !` does not permit being left uninitialized
    --> /home/sfackler/code/rstack/target/debug/build/dw-systest-b765583e4de77b6a/out/all.rs:7830:34
     |
7830 |                   let mut y: U = uninitialized();
     |                                  ^^^^^^^^^^^^^^^
     |                                  |
     |                                  this code causes undefined behavior when executed
     |                                  help: use `MaybeUninit<T>` instead
     |
     = note: Function pointers must be non-null

Add repository topics

I'll probably forget that this exists by the time I'll need it so let's make this repository more discoverable :)

Getting `internal error: entered unreachable code`

Hi,
I tried to follow the Readme and I'm getting into an unreachable in the lib.
I drilled it down to just having:
let mut cfg = ctest::TestGenerator::new().generate("../secp256k1-sys/src/lib.rs", "all.rs");

Full backtrace:

--- stderr
thread 'main' panicked at 'internal error: entered unreachable code', /home/elichai2/.cargo/registry/src/github.com-1ecc6299db9ec823/syntex_syntax2-0.0.2/src/ext/expand.rs:314:21
stack backtrace:
   0:     0x5653ecfbfac4 - backtrace::backtrace::libunwind::trace::h90669f559fb267f0
                               at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/libunwind.rs:88
   1:     0x5653ecfbfac4 - backtrace::backtrace::trace_unsynchronized::hffde4e353d8f2f9a
                               at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/mod.rs:66
   2:     0x5653ecfbfac4 - std::sys_common::backtrace::_print_fmt::heaf44068b7eaaa6a
                               at src/libstd/sys_common/backtrace.rs:77
   3:     0x5653ecfbfac4 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h88671019cf081de2
                               at src/libstd/sys_common/backtrace.rs:59
   4:     0x5653ecfdf16c - core::fmt::write::h4e6a29ee6319c9fd
                               at src/libcore/fmt/mod.rs:1052
   5:     0x5653ecfbbc87 - std::io::Write::write_fmt::hf06b1c86d898d7d6
                               at src/libstd/io/mod.rs:1426
   6:     0x5653ecfc1de5 - std::sys_common::backtrace::_print::h404ff5f2b50cae09
                               at src/libstd/sys_common/backtrace.rs:62
   7:     0x5653ecfc1de5 - std::sys_common::backtrace::print::hcc4377f1f882322e
                               at src/libstd/sys_common/backtrace.rs:49
   8:     0x5653ecfc1de5 - std::panicking::default_hook::{{closure}}::hc172eff6f35b7f39
                               at src/libstd/panicking.rs:204
   9:     0x5653ecfc1ad1 - std::panicking::default_hook::h7a68887d113f8029
                               at src/libstd/panicking.rs:224
  10:     0x5653ecfc244a - std::panicking::rust_panic_with_hook::hb7ad5693188bdb00
                               at src/libstd/panicking.rs:472
  11:     0x5653ece3dbeb - std::panicking::begin_panic::h5895e03812433dcb
                               at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447/src/libstd/panicking.rs:399
  12:     0x5653ece0b222 - syntex_syntax2::ext::expand::MacroExpander::expand::h18253f4574feeea7
                               at /home/elichai2/.cargo/registry/src/github.com-1ecc6299db9ec823/syntex_syntax2-0.0.2/src/ext/expand.rs:314
  13:     0x5653ece0a0f3 - syntex_syntax2::ext::expand::MacroExpander::expand_crate::h38981502652bb841
                               at /home/elichai2/.cargo/registry/src/github.com-1ecc6299db9ec823/syntex_syntax2-0.0.2/src/ext/expand.rs:219
  14:     0x5653ec92af37 - ctest::TestGenerator::_generate_files::h25ce2f5aaed48e73
                               at /home/elichai2/.cargo/registry/src/github.com-1ecc6299db9ec823/ctest-0.2.22/src/lib.rs:921
  15:     0x5653ec92a094 - ctest::TestGenerator::generate_files::h7c4c182d6f8b89a1
                               at /home/elichai2/.cargo/registry/src/github.com-1ecc6299db9ec823/ctest-0.2.22/src/lib.rs:870
  16:     0x5653ec929315 - ctest::TestGenerator::_generate::hd941ddc8a4908095
                               at /home/elichai2/.cargo/registry/src/github.com-1ecc6299db9ec823/ctest-0.2.22/src/lib.rs:806
  17:     0x5653ec925a42 - ctest::TestGenerator::generate::he1740da48b9120d1
                               at /home/elichai2/.cargo/registry/src/github.com-1ecc6299db9ec823/ctest-0.2.22/src/lib.rs:802
  18:     0x5653ec9258d6 - build_script_build::main::h8987232324dcabc9
                               at /home/elichai2/gits/rust-secp256k1/systest/build.rs:4
  19:     0x5653ec9259b0 - std::rt::lang_start::{{closure}}::h377f2547084926bb
                               at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447/src/libstd/rt.rs:67
  20:     0x5653ecfc1f13 - std::rt::lang_start_internal::{{closure}}::hb26e39676675046f
                               at src/libstd/rt.rs:52
  21:     0x5653ecfc1f13 - std::panicking::try::do_call::he4701ab6e48d80c0
                               at src/libstd/panicking.rs:305
  22:     0x5653ecfca357 - __rust_maybe_catch_panic
                               at src/libpanic_unwind/lib.rs:86
  23:     0x5653ecfc28f0 - std::panicking::try::hd3de25f3cb7024b8
                               at src/libstd/panicking.rs:281
  24:     0x5653ecfc28f0 - std::panic::catch_unwind::h86c02743a24e3d92
                               at src/libstd/panic.rs:394
  25:     0x5653ecfc28f0 - std::rt::lang_start_internal::h9cf8802361ad86c2
                               at src/libstd/rt.rs:51
  26:     0x5653ec925989 - std::rt::lang_start::h54f9b0f6e32ccd40
                               at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447/src/libstd/rt.rs:67
  27:     0x5653ec92593a - main
  28:     0x7fc919a65023 - __libc_start_main
  29:     0x5653ec92517e - _start
  30:                0x0 - <unknown>

Fails to build on nightly

The build (for ssh2's tests) is failing on nightly rust because of the extprim transitive dep via the unmaintained syntex crate. Can this dependency be replaced?

error: legacy asm! syntax is no longer supported
   --> C:\Users\wez\.cargo\registry\src\github.com-1ecc6299db9ec823\extprim-1.7.0\src\u128.rs:884:9
    |
884 |           asm!("
    |           ^---
    |           |
    |  _________help: replace with: `llvm_asm!`
    | |
885 | |             movq $2, %rax
886 | |             mulq $3
887 | |             movq %rax, $0
...   |
891 | |         : "r"(left), "r"(right)
892 | |         : "rax", "rdx");
    | |________________________^

error: aborting due to previous error

error: could not compile `extprim`.

Error with rust-openssl and Alpine Linux

Hey, the following Dockerfile:

FROM alpine
RUN apk add git cargo pkgconfig openssl openssl-dev
RUN git clone https://github.com/sfackler/rust-openssl /build
RUN cd /build && cargo build --release

produces the following error:

error: failed to run custom build command for `systest v0.1.0 (/build/systest)`

Caused by:
  process didn't exit successfully: `/build/target/release/build/systest-9a76ac7552bababf/build-script-build` (exit code: 101)
--- stderr
thread 'main' panicked at 'unknown os/family width: x86_64-alpine-linux-musl', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/ctest-0.2.22/src/lib.rs:1117:9

Is it correct that this panic comes from ctest?

typedef for a struct is missing the struct keyword

Him
if you have rust code like:

type flock64 = flock;
struct flock {...}

The C code will use flock64 instead of struct flock64.
Is that something I should handle with type_name myself?

CC rust-lang/libc#1561

Casts in fixed-size array lengths break ctest

thread 'main' panicked at 'unknown expr: expr(4294967295: SSL_MAX_MASTER_KEY_LENGTH as usize)', /home/sfackler/.cargo/registry/src/github.com-1ecc6299db9ec823/ctest-0.1.1/src/lib.rs:1195

Migrate from appveyor to azure pipelines

We should do this. Help wanted.

Test extern statics

ctest currently ignores extern statics, but it can perform some basic checks like making sure addresses match between C and Rust. This will minimally ensure that the symbol actually exists.

structs don't survive macro expansion

I'm out of my depth debugging what's happening but I observe struct names not surviving the expand macros phase.

If I run:

let mut structs = StructFinder {
    structs: HashSet::new(),
};
visit::walk_crate(&mut structs, &krate);

before let krate = ecx.monotonic_expander().expand_crate(krate); I see both enum and struct names in structs.struct, but when it runs after I see only enum names in structs.structs.

I'm using the stable compiler: rustc 1.23.0 (766bd11c8 2018-01-01)

Relies on deprecated syntex crate

Syntex is deprecated and unmaintained and has been so for two years. Can we find an alternative?

Support unions

This requires bumping syntex_syntax really far forward, and making those changes is a bit beyond my knowledge of libsyntax :(

builtin f32 type isn't handled

I'm running ctest on Rust bindings generated by bindgen, and while an older bindgen used c_float, newer versions use f32 instead. ctest happily converts c_float to float, but f32 yields errors like:

  cargo:warning=/Users/jdm/src/rust-harfbuzz/target/debug/build/harfbuzz-sys-test-256f53a27904a18b/out/all.c:2043:18: error: unknown type name 'f32'
  cargo:warning=                 f32* __test_field_type_hb_variation_t_value(struct hb_variation_t* b) {

Building with `-Werror` breaks certain libraries

This source line enables -Werror when building. This breaks the libz-sys crate's build process.

Is there any reason for enabling -Werror? Should it be made optional? Fixing the original C libz is another possibility, although that is a harder path.

README says documentation has example, it doesn't

Just trying to figure out what ctest does and does not do and if it might be something I could integrate into a few projects to help test FFI. The documentation is quite sparse at the moment.

Testing rust-libc (ctest with syntex-syntax 0.3.0 ?)

Is it possible to use ctest with syntex-syntax 0.3.0?

The reason I ask is because we are trying to bootstrap rust-libc, including running the tests (Note: it already works without the tests).

Now rust-libc requires rust-ctest for the tests and rust-ctest requires rust-gcc and rust-syntex-syntax. However, current rust-syntex-syntax has quite a large number of dependencies, including rust-libc (!). But rust-syntex-syntax 0.3.0 has only one dependency, rust-bitflags, which has itself none.

Therefore we would like to bootstrap current rust-libc using ctest with rust-syntex-syntax 0.3.0.

Also, does rust-syntex-syntax work with stable Rust at all?

So far I didn't get syntex-syntax 0.3.0 to work (lots of things broken) - however I thought before I try to fix it I'd ask whether it should be possible to use it (with reasonable effort).

Alternatively, is there another way to test rust-libc?

Our goal is to test rust-libc using a minimal amount of extra packages (whose artifacts we throw away after each libc test).

the cfg example is unclear

The example body states: https://docs.rs/ctest/0.1.7/ctest/struct.TestGenerator.html#method.cfg

The k argument is the #[cfg] value to define, and v is an optional value for differentiating between #[cfg(foo)] and #[cfg(foo = "bar")].

No idea what this exactly means (the example below uses different names). I suppose if I write k = "foo" and v = None then I get #[cfg(foo)], and if I write k = "foo" and v = Some("bar") I get #[cfg(foo = "bar")].

Is this correct?

The following example is shown:

let mut cfg = TestGenerator::new();
cfg.cfg("foo", None)
   .cfg("bar", Some("baz"));

but I have no idea what it is trying to tell me. Maybe add some comments next to each line, stating what the consequences of each .cfg invocations are?

Alignment checks are broken for u64 on i686-unknown-linux-gnu

Probably a conflation between preferred and minimal alignment? First seen in rust-openssl builds: https://circleci.com/gh/sfackler/rust-openssl/15593

Minimal example:

build.rs

fn main() {
    ctest::TestGenerator::new()
        .header("foo.h")
        .include("src")
        .generate("src/foo.rs", "all.rs");
}

src/foo.h

#define SHA_LONG64 unsigned long long

src/foo.rs

pub type SHA_LONG64 = u64;

src/main.rs

use libc::*;

use crate::foo::*;

mod foo;

include!(concat!(env!("OUT_DIR"), "/all.rs"));

$ cargo run --target i686-unknown-linux-gnu
   Compiling foo v0.1.0 (/home/sfackler/foo)
warning: unused import: `libc::*`
 --> src/main.rs:1:5
  |
1 | use libc::*;
  |     ^^^^^^^
  |
  = note: #[warn(unused_imports)] on by default

warning: type `SHA_LONG64` should have an upper camel case name
 --> src/foo.rs:1:10
  |
1 | pub type SHA_LONG64 = u64;
  |          ^^^^^^^^^^ help: convert the identifier to upper camel case: `ShaLong64`
  |
  = note: #[warn(non_camel_case_types)] on by default

warning: unused macro definition
  --> /home/sfackler/foo/target/i686-unknown-linux-gnu/debug/build/foo-0432c371df727992/out/all.rs:58:13
   |
58 | /             macro_rules! offset_of {
59 | |                 ($ty:ident, $field:ident) => (
60 | |                     (&((*(0 as *const $ty)).$field)) as *const _ as u64
61 | |                 )
62 | |             }
   | |_____________^
   |
   = note: #[warn(unused_macros)] on by default

    Finished dev [unoptimized + debuginfo] target(s) in 0.45s
     Running `target/i686-unknown-linux-gnu/debug/foo`
RUNNING ALL TESTS
bad SHA_LONG64 align: rust: 4 (0x4) != c 8 (0x8)
thread 'main' panicked at 'some tests failed', /home/sfackler/foo/target/i686-unknown-linux-gnu/debug/build/foo-0432c371df727992/out/all.rs:13:21
note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

References in C FFI should emit the restrict qualifier in the tests

When a C API has a restrict pointer, int foo(int * restrict); the pointer must not alias any other pointer. In the Rust side, this means that a raw *mut c_int pointer is not enough, and a &mut c_int must be used. Otherwise, the API is "unsound", as in, it allows passing it aliasing pointers.

However, some of these APIs do not require the pointer to point to initialize memory, that is, passing them a pointer to an uninitialized value is "ok" for C. Doing that via &mut c_int might be UB, and we might need to generate "something else" here.

This came up when validating the FFI wrapper in libc of lio_listio in FreeBSD, which takes a *const noalias *mut noalias T. Right now, we use *mut T and call it a day, but when validating the bindings, clang rightfully complains that our type is not compatible with the C type because it is missing the restrict qualifier. That is, we would be calling a function type that requires noalias from a prototype that does not, which is unsound.

AFAICT there is no way to express this from Rust. The first step here would be to start using &/&mut since that would make the API sound, even though that would forbid passing pointers to uninitialized memory.

cc @ralfj @Centril

panic on latest nightly

While running on the libgit2-sys crate, I'm getting a panic:

thread 'main' panicked at 'attempted to leave type extern "C" fn(u32, *const i8, *const libgit2_sys::git_diff_file, *const libgit2_sys::git_diff_file, *const libgit2_sys::git_diff_file, *mut core::ffi::c_void) -> i32 uninitialized, which is invalid', /rustc/c20d7eecbc0928b57da8fe30b2ef8528e2bdd5be/src/libcore/mem/mod.rs:536:5

This is caused by rust-lang/rust#66059. I'm guessing ctest will need to be rewritten to use MaybeUninit?

rustc 1.43.0-nightly (c20d7eecb 2020-03-11)

Warnings about deprecated atomic initializers

The generated code produces warnings about the use of deprecated atomic initializers:

warning: use of deprecated item 'std::sync::atomic::ATOMIC_BOOL_INIT': the `new` function is now preferred
 --> D:\a\libz-sys\libz-sys\target\x86_64-pc-windows-gnu\debug\build\systest-47d2caa40185cb8b\out/all.rs:4:49
  |
4 |             use std::sync::atomic::{AtomicBool, ATOMIC_BOOL_INIT, Ordering};
  |                                                 ^^^^^^^^^^^^^^^^
  |
  = note: `#[warn(deprecated)]` on by default

warning: use of deprecated item 'std::sync::atomic::ATOMIC_USIZE_INIT': the `new` function is now preferred
 --> D:\a\libz-sys\libz-sys\target\x86_64-pc-windows-gnu\debug\build\systest-47d2caa40185cb8b\out/all.rs:5:50
  |
5 |             use std::sync::atomic::{AtomicUsize, ATOMIC_USIZE_INIT};
  |                                                  ^^^^^^^^^^^^^^^^^

warning: use of deprecated item 'std::sync::atomic::ATOMIC_BOOL_INIT': the `new` function is now preferred
  --> D:\a\libz-sys\libz-sys\target\x86_64-pc-windows-gnu\debug\build\systest-47d2caa40185cb8b\out/all.rs:41:41
   |
41 |             static FAILED: AtomicBool = ATOMIC_BOOL_INIT;
   |                                         ^^^^^^^^^^^^^^^^ help: replace the use of the deprecated item: `AtomicBool::new(false)`

warning: use of deprecated item 'std::sync::atomic::ATOMIC_USIZE_INIT': the `new` function is now preferred
  --> D:\a\libz-sys\libz-sys\target\x86_64-pc-windows-gnu\debug\build\systest-47d2caa40185cb8b\out/all.rs:42:42
   |
42 |             static NTESTS: AtomicUsize = ATOMIC_USIZE_INIT;
   |                                          ^^^^^^^^^^^^^^^^^ help: replace the use of the deprecated item: `AtomicUsize::new(0)`

Not clear how to test `const <type> <obj>;` fields

See this comment in git2-rs for details: https://github.com/alexcrichton/git2-rs/pull/315#issuecomment-393006101

cannot parse modern rust due to reliance of syntex_syntax

Since ctest is using a ~2 year old version of libsyntax, it cannot handle any language changes since then. Are there any plans or ideas on how to move forward? I noticed you looking at this at dtolnay/syn#588, but I was wondering if you've found anything since.