zowens / crc32c Goto Github PK

View Code? Open in Web Editor NEW

41.0 41.0 19.0 78 KB

Fast CRC-32-Castagnoli implementation in Rust

Rust 100.00%

crc32 hashing rust rust-library

crc32c's People

Contributors

Stargazers

Watchers

Forkers

yjh0502 jasonlor my3157 vsrs nacardin csonto pqrstuvwx brooooooklyn nissaofthesea crisidev clbarnes cblichmann rukai calmofthestorm lizhanhui jayvdb nilismith paolobarbolini

crc32c's Issues

0.6.4 release?

Now that the new hasher traits have been added, is it possible to do a 0.6.4 release?

Thank you!

UB in build script

the build script creates uninitialized integers in a few places:

crc32c/build.rs

Line 13 in 47c2999

    
           let mut table: [[u32; 256]; 8] = unsafe { mem::MaybeUninit::uninit().assume_init() };

crc32c/build.rs

Lines 43 to 49 in 47c2999

    
           pub struct Matrix([u32; 32]); 
        
           impl Matrix { 
        
               /// Allocates space for a new matrix. 
        
               fn new() -> Self { 
        
                   unsafe { mem::MaybeUninit::uninit().assume_init() } 
        
               }

crc32c/build.rs

Line 135 in 47c2999

    
           let mut zeroes: [[u32; 256]; 4] = unsafe { mem::MaybeUninit::uninit().assume_init() };

afaik, this is undefined behavior (excerpt from MaybeUninit type docs)

Moreover, uninitialized memory is special in that it does not have a fixed value (“fixed” meaning “it won’t change without being written to”). Reading the same uninitialized byte multiple times can give different results. This makes it undefined behavior to have uninitialized data in a variable even if that variable has an integer type, which otherwise can hold any fixed bit pattern:
use std::mem::{self, MaybeUninit};

let x: i32 = unsafe { mem::uninitialized() }; // undefined behavior! ⚠️
// The equivalent code with `MaybeUninit<i32>`:
let x: i32 = unsafe { MaybeUninit::uninit().assume_init() }; // undefined behavior! ⚠️

i haven't looked at the code in too much detail but this may be an issue if there's references to this data before it's been initialized for reals.

incorrect use of unsafe

Most of the functions in hw.rs either need to be marked unsafe, or guarateed by the type system to be safe. Neither are true today.

From the std::arch docs:

The CPU the program is currently running on supports the function being called. For example it is unsafe to call an AVX2 function on a CPU that doesn't actually support AVX2.

The simplest way that I can tell to fix this in your code is to create a type that can only be constructed when the correct target features are enabled. e.g.,

struct SSE42(());

impl SSE42 {
    fn new() -> Option<SSE42> {
        if is_x86_feature_detected!("sse4.2") {
            Some(SSE42(()))
        } else {
            None
        }
    }
}

Then any function needing to use SSE4.2 can require a SSE42 parameter. And your crc32c_append function then looks like this:

pub fn crc32c_append(crc: u32, data: &[u8]) -> u32 {
    if let Some(receipt) = SSE42::new() {
        hw::crc32c(receipt, crc, data)
    } else {
        sw::crc32c(crc, data)
    }
}

The reason why the code today is wrong is because any user of hw.rs can call hw::crc32c as a safe function without actually verifying the assumptions required by the use of unsafe inside hw::crc32c.

Now, it is true that this is not exposed in the public crate API, which is good. But internally, this is still a violation of unsafe.

Maintain a CHANGELOG file

thank you for this crate!

would it be possible to create and maintain a CHANGELOG file to track changes between versions

Set a description / topics for the repository

Currently the GitHub main page says

No description, website, or topics provided.

Perhaps a small one-liner such as "Fast CRC-32-Castagnoli implementation in Rust".
Topics could be "crc32", "rust", and such.

Doesn't build on x86

master and 0.4.0 from Crates fail:

   Compiling crc32c v0.4.0 (/tmp/crc32c)
error[E0432]: unresolved import `std::arch::x86_64`
 --> src/hw.rs:5:5
  |
5 | use std::arch::x86_64 as simd;
  |     ^^^^^^^^^^^^^^^^^^^^^^^^^ no `x86_64` in `arch`

error: aborting due to previous error

For more information about this error, try `rustc --explain E0432`.
error: could not compile `crc32c`.

To learn more, run the command again with --verbose.

uname -a:

Linux nabtop 5.4.0-3-686-pae #1 SMP Debian 5.4.13-1 (2020-01-19) i686 GNU/Linux

rustc --version:

rustc 1.41.0 (5e1a79984 2020-01-27)

Build crc32c for ARM

I try to build crc32c for armv5te. I am getting an error:

error[E0658]: macro is_x86_feature_detected! is unstable
  --> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/crc32c-0.4.0/src/lib.rs:38:8
   |
38 |     if is_x86_feature_detected!("sse4.2") {
   |        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: for more information, see https://github.com/rust-lang/rust/issues/27731
   = help: add #![feature(stdsimd)] to the crate attributes to enable

error: 
        is_x86_feature_detected can only be used on x86 and x86_64 targets.
        You can prevent it from being used in other architectures by
        guarding it behind a cfg(target_arch) as follows:

            #[cfg(any(target_arch = "x86", target_arch = "x86_64"))] {
                if is_x86_feature_detected(...) { ... }
            }
        
  --> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/crc32c-0.4.0/src/lib.rs:38:8
   |
38 |     if is_x86_feature_detected!("sse4.2") {
   |        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: this error originates in a macro outside of the current crate (in Nightly builds, run with -Z external-macro-backtrace for more info)

error[E0432]: unresolved import `std::arch::x86_64`
 --> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/crc32c-0.4.0/src/hw.rs:5:5
  |
5 | use std::arch::x86_64 as simd;
  |     ^^^^^^^^^^^^^^^^^^^^^^^^^ no `x86_64` in `arch`

error: the feature named `sse4.2` is not valid for this target
  --> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/crc32c-0.4.0/src/hw.rs:72:18
   |
72 | #[target_feature(enable = "sse4.2")]
   |                  ^^^^^^^^^^^^^^^^^ `sse4.2` is not valid for this target

error: the feature named `sse4.2` is not valid for this target
  --> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/crc32c-0.4.0/src/hw.rs:78:18
   |
78 | #[target_feature(enable = "sse4.2")]
   |                  ^^^^^^^^^^^^^^^^^ `sse4.2` is not valid for this target

error: aborting due to 5 previous errors

Some errors have detailed explanations: E0432, E0658.
For more information about an error, try `rustc --explain E0432`.
error: Could not compile `crc32c`.
warning: build failed, waiting for other jobs to finish...
error: build failed

Crc32cHasher initial value should apply effect of refin

Refer https://reveng.sourceforge.io/crc-catalogue/all.htm and https://github.com/mrhooray/crc-rs/blob/7e4b1b4879e3bb881f150b550f150df1ff199f04/src/crc32.rs#L60

The initial value should consider refin = true.

Hasher Trait

It would be a useful ergonomic improvement to add the "hasher" trait to this crate to allow crc32c of structures for fast/quick validations. Thanks!

Software crc32c produces wrong results on big endian CPUs

I am trying to run some code which uses crc32c on powerpc 32 and 64 bit and I noticed it produces incorrect results.

I have tried to run the test suite of this crate using cross and they indeed fail:

❯❯❯ ~/github/crc32c ❯ cross test --target powerpc-unknown-linux-gnu
warning: unused manifest key: target.aarch64-unknown-linux-gnu.linker
    Finished test [unoptimized + debuginfo] target(s) in 0.15s
     Running unittests src/lib.rs (/target/powerpc-unknown-linux-gnu/debug/deps/crc32c-06d60a5bbd004a3e)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running tests/simple.rs (/target/powerpc-unknown-linux-gnu/debug/deps/simple-63febc81630ac4ff)

running 6 tests
test crc ... ok
test crc_append ... ok
test crc_combine ... FAILED
test long_string ... FAILED
test very_big ... FAILED
test very_small ... ok

failures:

---- crc_combine stdout ----
thread 'main' panicked at 'assertion failed: `(left == right)`
  left: `689410915`,
 right: `3475076554`', tests/simple.rs:25:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- long_string stdout ----
thread 'main' panicked at 'assertion failed: `(left == right)`
  left: `550182489`,
 right: `372323125`', tests/simple.rs:55:5

---- very_big stdout ----
thread 'main' panicked at 'assertion failed: `(left == right)`
  left: `314413457`,
 right: `1410821608`', tests/simple.rs:63:5


failures:
    crc_combine
    long_string
    very_big

test result: FAILED. 3 passed; 3 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.03s

error: test failed, to rerun pass '--test simple'

❯❯❯ ~/github/crc32c ❯ cross test --target powerpc64-unknown-linux-gnu
warning: unused manifest key: target.aarch64-unknown-linux-gnu.linker
    Finished test [unoptimized + debuginfo] target(s) in 0.14s
     Running unittests src/lib.rs (/target/powerpc64-unknown-linux-gnu/debug/deps/crc32c-d3c4704dfac2e462)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running tests/simple.rs (/target/powerpc64-unknown-linux-gnu/debug/deps/simple-7936969e39a66f78)

running 6 tests
test crc ... ok
test crc_append ... ok
test crc_combine ... FAILED
test long_string ... FAILED
test very_big ... FAILED
test very_small ... ok

failures:

---- crc_combine stdout ----
thread 'main' panicked at 'assertion failed: `(left == right)`
  left: `2861399345`,
 right: `3645618169`', tests/simple.rs:25:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- long_string stdout ----
thread 'main' panicked at 'assertion failed: `(left == right)`
  left: `550182489`,
 right: `372323125`', tests/simple.rs:55:5

---- very_big stdout ----
thread 'main' panicked at 'assertion failed: `(left == right)`
  left: `314413457`,
 right: `1410821608`', tests/simple.rs:63:5


failures:
    crc_combine
    long_string
    very_big

test result: FAILED. 3 passed; 3 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.04s

error: test failed, to rerun pass '--test simple'

Why ARM hardware acceleration is guarded by nightly

Is there anything that blocks ARM hardware acceleration feature from being stable?

Will it be better to guard it using a feature?

Fail to build on macOS M2

error[E0658]: use of unstable library feature 'stdsimd'
  --> /Users/tison/.cargo/registry/src/index.crates.io-6f17d22bba15001f/crc32c-0.6.5/src/hw_aarch64.rs:50:33
   |
50 |         .fold(crc, |crc, &next| simd::__crc32cb(crc, next))
   |                                 ^^^^^^^^^^^^^^^
   |
   = note: see issue #48556 <https://github.com/rust-lang/rust/issues/48556> for more information
   = help: add `#![feature(stdsimd)]` to the crate attributes to enable

error[E0658]: use of unstable library feature 'stdsimd'
  --> /Users/tison/.cargo/registry/src/index.crates.io-6f17d22bba15001f/crc32c-0.6.5/src/hw_aarch64.rs:62:5
   |
62 |     simd::__crc32cd(crc, next)
   |     ^^^^^^^^^^^^^^^
   |
   = note: see issue #48556 <https://github.com/rust-lang/rust/issues/48556> for more information
   = help: add `#![feature(stdsimd)]` to the crate attributes to enable

error[E0635]: unknown feature `stdarch_arm_crc32`
  --> /Users/tison/.cargo/registry/src/index.crates.io-6f17d22bba15001f/crc32c-0.6.5/src/lib.rs:23:60
   |
23 | #![cfg_attr(all(target_arch = "aarch64", nightly), feature(stdarch_arm_crc32))]
   |                                                            ^^^^^^^^^^^^^^^^^

Some errors have detailed explanations: E0635, E0658.
For more information about an error, try `rustc --explain E0635`.
error: could not compile `crc32c` (lib) due to 3 previous errors

$ rustc --version    
rustc 1.76.0-nightly (3f28fe133 2023-12-18)
$ uname -a
Darwin tisondeMacBook-Pro.local 22.5.0 Darwin Kernel Version 22.5.0: Mon Apr 24 20:53:19 PDT 2023; root:xnu-8796.121.2~5/RELEASE_ARM64_T6020 arm64

UB when `util::split` returns zero-length slices

util::split documents the following invariants of the returned slices:

crc32c/src/util.rs

Lines 3 to 6 in 47c2999

    
           /// Splits a buffer into three subslices: 
        
           /// - the first one is up to the first 8-byte aligned address. 
        
           /// - the second one is 8-byte aligned and its length is a multiple of 8. 
        
           /// - the third one is 8-byte aligned but its length is less than 8.

however, mid doesn't seem to uphold these.

adding the following to the function before it returns (start, mid, end) results in failed assertions.

// manual implementation of unstable `pointer::is_aligned_to`
#[cfg(debug_assertions)]
fn is_aligned_to<T>(ptr: *const T, align: usize) -> bool {
    assert!(align.is_power_of_two());
    ptr as usize % align == 0
}

/* `mid` invariants */
debug_assert!(is_aligned_to(mid.as_ptr(), 8)); // 8-byte aligned
debug_assert!(mid.len() % 8 == 0); // len is multiple of 8

/* `end` invariants */
debug_assert!(is_aligned_to(end.as_ptr(), 8)); // 8-byte aligned
debug_assert!(end.len() < 8); // len is less than 8

$ cargo test
<cut>
running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running tests/simple.rs (target/debug/deps/simple-561134e7e9ea1232)

running 6 tests
test crc ... ok
test crc_append ... ok
test long_string ... FAILED
test crc_combine ... FAILED
test very_small ... FAILED
test very_big ... ok

failures:

---- long_string stdout ----
thread 'long_string' panicked at 'assertion failed: mid.len() % 8 == 0', src/util.rs:48:5

---- crc_combine stdout ----
thread 'crc_combine' panicked at 'assertion failed: is_aligned_to(mid.as_ptr(), 8)', src/util.rs:47:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- very_small stdout ----
thread 'very_small' panicked at 'assertion failed: is_aligned_to(mid.as_ptr(), 8)', src/util.rs:47:5


failures:
    crc_combine
    long_string
    very_small

test result: FAILED. 3 passed; 3 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

error: test failed, to rerun pass '--test simple'

if there aren't any issues with my tests, then this may be unsound, since we're creating a &[u64] with unaligned elements.

EDIT: my tests were a little messed up but they're fixed now, assertions still fail

Set up continuous integration

Travis provides free CI for open-source projects. They also have a guide for setting it up.

A simple .travis.yml could be

language: rust
rust:
  - nightly # Require nightly, since we need some unstable features.
cache: cargo

Nightly build fails due to removal of the unstable `stdsimd` feature

The latest nightly breaks this crate on this line because the unstable stdsimd feature has been removed:

crc32c/src/lib.rs

Line 23 in d70d4e2

#![cfg_attr(nightly, feature(stdsimd))]

The commit removing stdsimd is rust-lang/rust@ea37e80
Relevant tracking issues? rust-lang/rust#27731 and rust-lang/rust#48556

	pub struct Matrix([u32; 32]);

	impl Matrix {
	/// Allocates space for a new matrix.
	fn new() -> Self {
	unsafe { mem::MaybeUninit::uninit().assume_init() }
	}

	/// Splits a buffer into three subslices:
	/// - the first one is up to the first 8-byte aligned address.
	/// - the second one is 8-byte aligned and its length is a multiple of 8.
	/// - the third one is 8-byte aligned but its length is less than 8.

zowens / crc32c Goto Github PK

crc32c's People

Contributors

Stargazers

Watchers

Forkers

crc32c's Issues

Recommend Projects

Recommend Topics

Recommend Org