10xgenomics / lz4-rs Goto Github PK

View Code? Open in Web Editor NEW

This project forked from bozaro/lz4-rs

42.0 42.0 20.0 3 MB

Rust LZ4 bindings

License: MIT License

Rust 100.00%

lz4-rs's People

Contributors

Stargazers

Watchers

Forkers

alecmocatta sumibi-yakitori roncohen analyzedfir agile86 janpaul123 alyoshavasilieva redvoxinc joshlind aptos-labs zyguan bchocho tisonkun bgianfo kastixx thehamsta musicinmybrain higumachan drsloth

lz4-rs's Issues

Any plan to update to lz4 v1.9.4

Thanks for the great crate! lz4 v1.9.4 has been released a few days ago. There is a notably performance improvement on high-end ARM64 platform. Any plan to update the lib?

Fail to cross compilation on Linux for Mac

Hi,

I'm working on cross-compilation of a project on Linux for Mac binaries. Got an error currently:

error: failed to run custom build command for `lz4-sys v1.9.2`

Caused by:
  process didn't exit successfully: `/opt/core/target/release/build/lz4-sys-635d53f76c21d05f/build-script-build` (exit status: 1)
  --- stdout
  TARGET = Some("aarch64-apple-darwin")
  HOST = Some("x86_64-unknown-linux-gnu")
  CC_aarch64-apple-darwin = None
  CC_aarch64_apple_darwin = None
  TARGET_CC = None
  CC = None
  CROSS_COMPILE = None
  CFLAGS_aarch64-apple-darwin = None
  CFLAGS_aarch64_apple_darwin = None
  TARGET_CFLAGS = None
  CFLAGS = None
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some("true")
  CARGO_CFG_TARGET_FEATURE = Some("aes,crc,dit,dotprod,dpb,dpb2,fcma,fhm,flagm,fp16,frintts,jsconv,llvm14-builtins-abi,lor,lse,neon,paca,pacg,pan,pmuv3,ras,rcpc,rcpc2,rdm,sb,sha2,sha3,ssbs,v8.1a,v8.2a,v8.3a,v8.4a,vh")
  running: "cc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-arch" "arm64" "-Wall" "-Wextra" "-o" "/opt/core/target/aarch64-apple-darwin/release/build/lz4-sys-6a23633e37c16f77/out/liblz4/lib/lz4.o" "-c" "liblz4/lib/lz4.c"
  cargo:warning=cc: error: arm64: No such file or directory
  cargo:warning=cc: error: unrecognized command line option ‘-arch’; did you mean ‘-march=’?
  exit status: 1
  running: "cc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-arch" "arm64" "-Wall" "-Wextra" "-o" "/opt/core/target/aarch64-apple-darwin/release/build/lz4-sys-6a23633e37c16f77/out/liblz4/lib/lz4frame.o" "-c" "liblz4/lib/lz4frame.c"
  cargo:warning=cc: error: arm64: No such file or directory
  cargo:warning=cc: error: unrecognized command line option ‘-arch’; did you mean ‘-march=’?
  exit status: 1

  --- stderr

So based on https://stackoverflow.com/questions/4391192/why-do-i-get-cc1plus-error-unrecognized-command-line-option-arch, seems -arch is only for Xcode's gcc. Not sure if lz4-sys's custom build script is able to make some change to fix that?

lz4-sys failing build in Windows

Hi, I've been using lz4-rs for a cross-platform project. The project runs fine on Linux. I'm on my way to make it compatible for Windows.

Compiling lz4-sys crate fails on Windows(x86_64-pc-windows-gnu) with the following message:

C:\path\to\project>cargo build
   Compiling lz4-sys v1.9.2
error: failed to run custom build command for `lz4-sys v1.9.2`

Caused by:
  process didn't exit successfully: `C:\path\to\project\target\debug\build\lz4-sys-eda498a4f8e49e77\build-script-build` (exit code: 1)
--- stdout
TARGET = Some("x86_64-pc-windows-gnu")
HOST = Some("x86_64-pc-windows-gnu")
CC_x86_64-pc-windows-gnu = None
CC_x86_64_pc_windows_gnu = None
HOST_CC = None
CC = None
CFLAGS_x86_64-pc-windows-gnu = None
CFLAGS_x86_64_pc_windows_gnu = None
HOST_CFLAGS = None
CFLAGS = None
CRATE_CC_NO_DEFAULTS = None
DEBUG = Some("true")
CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2")
running: "gcc.exe" "-O3" "-ffunction-sections" "-fdata-sections" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-o" "C:\\path\\to\\project\\target\\debug\\build\\lz4-sys-74730fdcc35cca62\\out\\liblz4/lib/lz4.o" "-c" "liblz4/lib/lz4.c"
exit code: 0
running: "gcc.exe" "-O3" "-ffunction-sections" "-fdata-sections" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-o" "C:\\path\\to\\project\\target\\debug\\build\\lz4-sys-74730fdcc35cca62\\out\\liblz4/lib/lz4frame.o" "-c" "liblz4/lib/lz4frame.c"
cargo:warning=during RTL pass: final
cargo:warning=liblz4/lib/lz4frame.c: In function 'LZ4F_compressBlockHC_continue':
cargo:warning=liblz4/lib/lz4frame.c:795:1: internal compiler error: in based_loc_descr, at dwarf2out.c:14264
cargo:warning= }
cargo:warning= ^
cargo:warning=libbacktrace could not find executable to open
cargo:warning=Please submit a full bug report,
cargo:warning=with preprocessed source if appropriate.
cargo:warning=See <https://sourceforge.net/projects/mingw-w64> for instructions.
exit code: 1

--- stderr


error occurred: Command "gcc.exe" "-O3" "-ffunction-sections" "-fdata-sections" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-o" "C:\\path\\to\\project\\target\\debug\\build\\lz4-sys-74730fdcc35cca62\\out\\liblz4/lib/lz4frame.o" "-c" "liblz4/lib/lz4frame.c" with args "gcc.exe" did not execute successfully (status code exit code: 1).



warning: build failed, waiting for other jobs to finish...
error: build failed

After this, I tried manually fetching the source for lz4-sys from this repo. Compiling it using gcc throws the same message.

C:\path\to\libs\lz4-sys>gcc -O3 -ffunction-sections -fdata-sections -g -fno-omit-frame-pointer -m64 -Wall -Wextra -o lz4frame.o -c lz4frame.c
during RTL pass: final
lz4frame.c: In function 'LZ4F_compressBlockHC_continue':
lz4frame.c:795:1: internal compiler error: in based_loc_descr, at dwarf2out.c:14264
 }
 ^
libbacktrace could not find executable to open
Please submit a full bug report,
with preprocessed source if appropriate.
See <https://sourceforge.net/projects/mingw-w64> for instructions.

With verbose flag (if it helps):

C:\path\to\libs\lz4-sys>gcc -O3 -ffunction-sections -fdata-sections -g -fno-omit-frame-pointer -m64 -Wall -Wextra -o lz4frame.o -c lz4frame.c --verbose
Using built-in specs.
COLLECT_GCC=D:\Core\lib\mingw\tools\install\mingw64\bin\gcc.exe
Target: x86_64-w64-mingw32
Configured with: ../../../src/gcc-8.1.0/configure --host=x86_64-w64-mingw32 --build=x86_64-w64-mingw32 --target=x86_64-w64-mingw32 --prefix=/mingw64 --with-sysroot=/c/mingw810/x86_64-810-posix-seh-rt_v6-rev0/mingw64 --enable-shared --enable-static --disable-multilib --enable-languages=c,c++,fortran,lto --enable-libstdcxx-time=yes --enable-threads=posix --enable-libgomp --enable-libatomic --enable-lto --enable-graphite --enable-checking=release --enable-fully-dynamic-string --enable-version-specific-runtime-libs --disable-libstdcxx-pch --disable-libstdcxx-debug --enable-bootstrap --disable-rpath --disable-win32-registry --disable-nls --disable-werror --disable-symvers --with-gnu-as --with-gnu-ld --with-arch=nocona --with-tune=core2 --with-libiconv --with-system-zlib --with-gmp=/c/mingw810/prerequisites/x86_64-w64-mingw32-static --with-mpfr=/c/mingw810/prerequisites/x86_64-w64-mingw32-static --with-mpc=/c/mingw810/prerequisites/x86_64-w64-mingw32-static --with-isl=/c/mingw810/prerequisites/x86_64-w64-mingw32-static --with-pkgversion='x86_64-posix-seh-rev0, Built by MinGW-W64 project' --with-bugurl=https://sourceforge.net/projects/mingw-w64 CFLAGS='-O2 -pipe -fno-ident -I/c/mingw810/x86_64-810-posix-seh-rt_v6-rev0/mingw64/opt/include -I/c/mingw810/prerequisites/x86_64-zlib-static/include -I/c/mingw810/prerequisites/x86_64-w64-mingw32-static/include' CXXFLAGS='-O2 -pipe -fno-ident -I/c/mingw810/x86_64-810-posix-seh-rt_v6-rev0/mingw64/opt/include -I/c/mingw810/prerequisites/x86_64-zlib-static/include -I/c/mingw810/prerequisites/x86_64-w64-mingw32-static/include' CPPFLAGS=' -I/c/mingw810/x86_64-810-posix-seh-rt_v6-rev0/mingw64/opt/include -I/c/mingw810/prerequisites/x86_64-zlib-static/include -I/c/mingw810/prerequisites/x86_64-w64-mingw32-static/include' LDFLAGS='-pipe -fno-ident -L/c/mingw810/x86_64-810-posix-seh-rt_v6-rev0/mingw64/opt/lib -L/c/mingw810/prerequisites/x86_64-zlib-static/lib -L/c/mingw810/prerequisites/x86_64-w64-mingw32-static/lib '
Thread model: posix
gcc version 8.1.0 (x86_64-posix-seh-rev0, Built by MinGW-W64 project)
COLLECT_GCC_OPTIONS='-O3' '-ffunction-sections' '-fdata-sections' '-g' '-fno-omit-frame-pointer' '-m64' '-Wall' '-Wextra' '-o' 'lz4frame.o' '-c' '-v' '-mtune=core2' '-march=nocona'
 D:/Core/lib/mingw/tools/install/mingw64/bin/../libexec/gcc/x86_64-w64-mingw32/8.1.0/cc1.exe -quiet -v -iprefix D:/Core/lib/mingw/tools/install/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.1.0/ -D_REENTRANT lz4frame.c -quiet -dumpbase lz4frame.c -m64 -mtune=core2 -march=nocona -auxbase-strip lz4frame.o -g -O3 -Wall -Wextra -version -ffunction-sections -fdata-sections -fno-omit-frame-pointer -o C:\Users\UTKARS~1\AppData\Local\Temp\cc3WERQo.s
GNU C17 (x86_64-posix-seh-rev0, Built by MinGW-W64 project) version 8.1.0 (x86_64-w64-mingw32)
        compiled by GNU C version 8.1.0, GMP version 6.1.2, MPFR version 4.0.1, MPC version 1.1.0, isl version isl-0.18-GMP

GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
ignoring duplicate directory "D:/Core/lib/mingw/tools/install/mingw64/lib/gcc/../../lib/gcc/x86_64-w64-mingw32/8.1.0/include"
ignoring nonexistent directory "C:/mingw810/x86_64-810-posix-seh-rt_v6-rev0/mingw64C:/msys64/mingw64/lib/gcc/x86_64-w64-mingw32/8.1.0/../../../../include"
ignoring duplicate directory "D:/Core/lib/mingw/tools/install/mingw64/lib/gcc/../../lib/gcc/x86_64-w64-mingw32/8.1.0/include-fixed"
ignoring duplicate directory "D:/Core/lib/mingw/tools/install/mingw64/lib/gcc/../../lib/gcc/x86_64-w64-mingw32/8.1.0/../../../../x86_64-w64-mingw32/include"
ignoring nonexistent directory "C:/mingw810/x86_64-810-posix-seh-rt_v6-rev0/mingw64/mingw/include"
#include "..." search starts here:
#include <...> search starts here:
 D:/Core/lib/mingw/tools/install/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.1.0/include
 D:/Core/lib/mingw/tools/install/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.1.0/include-fixed
 D:/Core/lib/mingw/tools/install/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.1.0/../../../../x86_64-w64-mingw32/include
End of search list.
GNU C17 (x86_64-posix-seh-rev0, Built by MinGW-W64 project) version 8.1.0 (x86_64-w64-mingw32)
        compiled by GNU C version 8.1.0, GMP version 6.1.2, MPFR version 4.0.1, MPC version 1.1.0, isl version isl-0.18-GMP

GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
Compiler executable checksum: 985ce7ae6dd3a696cd146ca9896b0035
during RTL pass: final
lz4frame.c: In function 'LZ4F_compressBlockHC_continue':
lz4frame.c:795:1: internal compiler error: in based_loc_descr, at dwarf2out.c:14264
 }
 ^
libbacktrace could not find executable to open
Please submit a full bug report,
with preprocessed source if appropriate.
See <https://sourceforge.net/projects/mingw-w64> for instructions.

Further searching for a way to make it work, I came across https://gcc.gnu.org/bugzilla/show_bug.cgi?id=49878

Surprisingly removing -fno-omit-frame-pointer does help and the compilation is successful.

C:\path\to\libs\lz4-sys>gcc -O3 -ffunction-sections -fdata-sections -g -m64 -Wall -Wextra -o lz4frame.o -c lz4frame.c --verbose
Using built-in specs.
COLLECT_GCC=D:\Core\lib\mingw\tools\install\mingw64\bin\gcc.exe
Target: x86_64-w64-mingw32
Configured with: ../../../src/gcc-8.1.0/configure --host=x86_64-w64-mingw32 --build=x86_64-w64-mingw32 --target=x86_64-w64-mingw32 --prefix=/mingw64 --with-sysroot=/c/mingw810/x86_64-810-posix-seh-rt_v6-rev0/mingw64 --enable-shared --enable-static --disable-multilib --enable-languages=c,c++,fortran,lto --enable-libstdcxx-time=yes --enable-threads=posix --enable-libgomp --enable-libatomic --enable-lto --enable-graphite --enable-checking=release --enable-fully-dynamic-string --enable-version-specific-runtime-libs --disable-libstdcxx-pch --disable-libstdcxx-debug --enable-bootstrap --disable-rpath --disable-win32-registry --disable-nls --disable-werror --disable-symvers --with-gnu-as --with-gnu-ld --with-arch=nocona --with-tune=core2 --with-libiconv --with-system-zlib --with-gmp=/c/mingw810/prerequisites/x86_64-w64-mingw32-static --with-mpfr=/c/mingw810/prerequisites/x86_64-w64-mingw32-static --with-mpc=/c/mingw810/prerequisites/x86_64-w64-mingw32-static --with-isl=/c/mingw810/prerequisites/x86_64-w64-mingw32-static --with-pkgversion='x86_64-posix-seh-rev0, Built by MinGW-W64 project' --with-bugurl=https://sourceforge.net/projects/mingw-w64 CFLAGS='-O2 -pipe -fno-ident -I/c/mingw810/x86_64-810-posix-seh-rt_v6-rev0/mingw64/opt/include -I/c/mingw810/prerequisites/x86_64-zlib-static/include -I/c/mingw810/prerequisites/x86_64-w64-mingw32-static/include' CXXFLAGS='-O2 -pipe -fno-ident -I/c/mingw810/x86_64-810-posix-seh-rt_v6-rev0/mingw64/opt/include -I/c/mingw810/prerequisites/x86_64-zlib-static/include -I/c/mingw810/prerequisites/x86_64-w64-mingw32-static/include' CPPFLAGS=' -I/c/mingw810/x86_64-810-posix-seh-rt_v6-rev0/mingw64/opt/include -I/c/mingw810/prerequisites/x86_64-zlib-static/include -I/c/mingw810/prerequisites/x86_64-w64-mingw32-static/include' LDFLAGS='-pipe -fno-ident -L/c/mingw810/x86_64-810-posix-seh-rt_v6-rev0/mingw64/opt/lib -L/c/mingw810/prerequisites/x86_64-zlib-static/lib -L/c/mingw810/prerequisites/x86_64-w64-mingw32-static/lib '
Thread model: posix
gcc version 8.1.0 (x86_64-posix-seh-rev0, Built by MinGW-W64 project)
COLLECT_GCC_OPTIONS='-O3' '-ffunction-sections' '-fdata-sections' '-g' '-m64' '-Wall' '-Wextra' '-o' 'lz4frame.o' '-c' '-v' '-mtune=core2' '-march=nocona'
 D:/Core/lib/mingw/tools/install/mingw64/bin/../libexec/gcc/x86_64-w64-mingw32/8.1.0/cc1.exe -quiet -v -iprefix D:/Core/lib/mingw/tools/install/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.1.0/ -D_REENTRANT lz4frame.c -quiet -dumpbase lz4frame.c -m64 -mtune=core2 -march=nocona -auxbase-strip lz4frame.o -g -O3 -Wall -Wextra -version -ffunction-sections -fdata-sections -o C:\Users\UTKARS~1\AppData\Local\Temp\ccOaxb3O.s
GNU C17 (x86_64-posix-seh-rev0, Built by MinGW-W64 project) version 8.1.0 (x86_64-w64-mingw32)
        compiled by GNU C version 8.1.0, GMP version 6.1.2, MPFR version 4.0.1, MPC version 1.1.0, isl version isl-0.18-GMP

GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
ignoring duplicate directory "D:/Core/lib/mingw/tools/install/mingw64/lib/gcc/../../lib/gcc/x86_64-w64-mingw32/8.1.0/include"
ignoring nonexistent directory "C:/mingw810/x86_64-810-posix-seh-rt_v6-rev0/mingw64C:/msys64/mingw64/lib/gcc/x86_64-w64-mingw32/8.1.0/../../../../include"
ignoring duplicate directory "D:/Core/lib/mingw/tools/install/mingw64/lib/gcc/../../lib/gcc/x86_64-w64-mingw32/8.1.0/include-fixed"
ignoring duplicate directory "D:/Core/lib/mingw/tools/install/mingw64/lib/gcc/../../lib/gcc/x86_64-w64-mingw32/8.1.0/../../../../x86_64-w64-mingw32/include"
ignoring nonexistent directory "C:/mingw810/x86_64-810-posix-seh-rt_v6-rev0/mingw64/mingw/include"
#include "..." search starts here:
#include <...> search starts here:
 D:/Core/lib/mingw/tools/install/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.1.0/include
 D:/Core/lib/mingw/tools/install/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.1.0/include-fixed
 D:/Core/lib/mingw/tools/install/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.1.0/../../../../x86_64-w64-mingw32/include
End of search list.
GNU C17 (x86_64-posix-seh-rev0, Built by MinGW-W64 project) version 8.1.0 (x86_64-w64-mingw32)
        compiled by GNU C version 8.1.0, GMP version 6.1.2, MPFR version 4.0.1, MPC version 1.1.0, isl version isl-0.18-GMP

GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
Compiler executable checksum: 985ce7ae6dd3a696cd146ca9896b0035
COLLECT_GCC_OPTIONS='-O3' '-ffunction-sections' '-fdata-sections' '-g' '-m64' '-Wall' '-Wextra' '-o' 'lz4frame.o' '-c' '-v' '-mtune=core2' '-march=nocona'
 D:/Core/lib/mingw/tools/install/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.1.0/../../../../x86_64-w64-mingw32/bin/as.exe -v --64 -o lz4frame.o C:\Users\UTKARS~1\AppData\Local\Temp\ccOaxb3O.s
GNU assembler version 2.30 (x86_64-w64-mingw32) using BFD version (GNU Binutils) 2.30
COMPILER_PATH=D:/Core/lib/mingw/tools/install/mingw64/bin/../libexec/gcc/x86_64-w64-mingw32/8.1.0/;D:/Core/lib/mingw/tools/install/mingw64/bin/../libexec/gcc/;D:/Core/lib/mingw/tools/install/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.1.0/../../../../x86_64-w64-mingw32/bin/
LIBRARY_PATH=D:/Core/lib/mingw/tools/install/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.1.0/;D:/Core/lib/mingw/tools/install/mingw64/bin/../lib/gcc/;D:/Core/lib/mingw/tools/install/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.1.0/../../../../x86_64-w64-mingw32/lib/../lib/;D:/Core/lib/mingw/tools/install/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.1.0/../../../../lib/;D:/Core/lib/mingw/tools/install/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.1.0/../../../../x86_64-w64-mingw32/lib/;D:/Core/lib/mingw/tools/install/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.1.0/../../../
COLLECT_GCC_OPTIONS='-O3' '-ffunction-sections' '-fdata-sections' '-g' '-m64' '-Wall' '-Wextra' '-o' 'lz4frame.o' '-c' '-v' '-mtune=core2' '-march=nocona'

I can see lzframe.o in the directory.

Now I'm not that well-versed with this gcc-option and I'm not sure if this is a solution to the problem or if I've simply overlooked the issue here.

I'm hoping this is the right place to open an issue following the change in project maintainers.

Thanks and cheers.

Unable to set content size

Currently, the ability to set the content size of the uncompressed data is not supported by this library, but it is supported in the C library: https://github.com/lz4/lz4/blob/d44371841a2f1728a3f36839fd4b7e872d0927d3/lib/lz4frame.h#L180

I've submitted a PR (#23) that exposes this functionality through lz4-sys and the lz4 Encoder struct.

I would like to work with you to get this PR accepted. If you have the time, could you please review the PR and let me know if anything needs to be changed or addressed?

Really appreciate you taking over this project and appreciate your (and others) hard work.

failed to compress and decompress empty bytes

The following code fails, which is not desired behavior

use lz4::block::{compress,decompress};
let v = vec![0u8; 0];
let comp_with_prefix = compress(&v, None, true).unwrap();
assert_eq!(v, decompress(&comp_with_prefix, None).unwrap());

(originally posted the issue here: bozaro#55)

RUSTFLAGS='-C strip=symbols' cargo build -p 'lib_project' --release --target aarch64-linux-android
RUSTFLAGS='-C strip=symbols' cargo build -p 'lib_project' --release --target armv7-linux-androideabi
code:

///compress lz4
  fn compress_lz4(bytes: &Vec<u8>, level: u32) -> anyhow::Result<Vec<u8>> {
        let output = LzBuffer{
            buffer: vec![0x2, 0, 0, 0, 0],
            wpos: 5,
        };
        let mut encoder = EncoderBuilder::new()
            .level(level)
            .build(output)?;

        let mut buffer = CompressInput{
            buffer: bytes,
            rpos: 0,
        };
        std::io::copy(&mut buffer, &mut encoder)?;
        let (mut output, r) = encoder.finish();
        r?;
        Ok(output.buffer)
}
///decompress lz4 bytes buffer    
fn decompress_lz4(bytes: &[u8], size: usize) -> anyhow::Result<Vec<u8>> {
      let mut decoder = Decoder::new(std::io::Cursor::new(bytes))?;
      let mut out_buffer = Vec::with_capacity(size);
      std::io::copy(&mut decoder, &mut out_buffer)?;
      Ok(out_buffer)
}

config:

[target.aarch64-linux-android]
ar = "NDK/aarch64/bin/aarch64-linux-android-ar"
linker = "NDK/aarch64/bin/aarch64-linux-android22-clang"

[target.armv7-linux-androideabi]
ar = "NDK/armv7/bin/arm-linux-androideabi-ar"
linker = "NDK/armv7/bin/armv7a-linux-androideabi22-clang"

build error:
error: failed to run custom build command for lz4-sys v1.9.3

Caused by:
process didn't exit successfully: /home/banagame/BattleServer/server/Rust/target/release/build/lz4-sys-5910c428c3e500ee/build-script-build (exit status: 1)
--- stdout
TARGET = Some("aarch64-linux-android")
HOST = Some("x86_64-unknown-linux-gnu")
CC_aarch64-linux-android = None
CC_aarch64_linux_android = None
TARGET_CC = None
CC = None
CFLAGS_aarch64-linux-android = None
CFLAGS_aarch64_linux_android = None
TARGET_CFLAGS = None
CFLAGS = None
CRATE_CC_NO_DEFAULTS = None
DEBUG = Some("false")
running: "aarch64-linux-android-clang" "-O3" "-DANDROID" "-ffunction-sections" "-fdata-sections" "-fPIC" "-Wall" "-Wextra" "-o" "/home/banagame/BattleServer/server/Rust/target/aarch64-linux-android/release/build/lz4-sys-f443fb51687be329/out/liblz4/lib/lz4.o" "-c" "liblz4/lib/lz4.c"

--- stderr

error occurred: Failed to find tool. Is aarch64-linux-android-clang installed?

Support `de/compress_into(src: &[u8], dst: &mut [u8])`

I'm in the situation where I would like to re-use buffers that are overallocated, and this API would be very helpful. Willing to attempt a contribution if possible; haven't looked at what changes the source code would need or how friendly the C interaction is.

Decompressed data does not match original

I wrote this program here:
`use std::io::{self, Write, Read};

fn main() {

// encode to a vec
let mut enc = lz4::EncoderBuilder::new().build(Vec::new()).unwrap();

// build our expected
let mut expected = vec![0; 65536];

// write 65530 bytes to our encoder and populate expected with same bytes
for i in 0..65530 {
    enc.write(&[(i % 128) as u8; 1]).unwrap();
    expected[i] = (i % 128) as u8;
}

// finish the encoder and check for error
let (v, r) = enc.finish();
r.unwrap();

// builder decoder around returned vec
let mut dec = lz4::Decoder::new(&v[..]).unwrap();

// compare decoded bytes to expected bytes
let mut got = vec![0; 65536];
dec.read_exact(&mut got[..]);
assert_eq!(expected, got);

It would be my expectation that the assert_eq!() call would come back as true, but panics due to expected and got not being equal. This program works when using other lz4 libraries. Am I misunderstanding what finish() is supposed to do, or is this a bug?

May I help in maintaining and releasing this crate?

Hi @pmarks!

It seems this repo has no releases from the last year, while there are still several inflight commits as well as pending issues/PRs. I'm willing to help in maintaining and releasing this crate - the original purpose is to release my patch #31. Do you have some suggestions how we can collaborate in this direction?

I may think of become a member of the org or pass the repo to my domain.

Support for block compression only

Hi,
Does this crate support block compression only?
So right now it uses block and stream as far as I know.
Kind regards

Significant decompression speed regression with rust 1.73 toolchain

I'm doing some benchmarks with a large (1.7GB) CSV file, and it seems that upgrading from rust 1.71 to 1.73 has caused a significant performance regression for lz4::Decoder.

Some results:

Rust 1.72

lz4 1...
[lz4 1] Compression: 20.3244%, Time: 2.16s
Decompress in: 1.25s
lz4 4...
[lz4 4] Compression: 15.5764%, Time: 12.56s
Decompress in: 1.10s
lz4 7...
[lz4 7] Compression: 15.1194%, Time: 40.19s
Decompress in: 1.11s

Rust 1.73

lz4 1...
[lz4 1] Compression: 20.3244%, Time: 2.15s
^C⏎

rs-test on  master [?] is 󰏗 v0.1.0 via  v1.73.0 took 11m

^ I killed the benchmark during decompression because it took over 11 minutes.

benchmark code

    for level in [1, 4, 7] {
        println!("lz4 {}...", level);
        let mut in_f = std::fs::File::open(in_f_path).unwrap();
        let start_size = in_f.metadata().unwrap().len();
        let now = Instant::now();
        let out_path = format!("out-{}.lz4", level);

        let out_f = std::fs::File::create(&out_path).unwrap();
        let mut encoder = EncoderBuilder::new().level(level).build(out_f).unwrap();
        std::io::copy(&mut in_f, &mut encoder).unwrap();
        let (output, result) = encoder.finish();
        if let Err(e) = result {
            println!("{:#?}", e);
        }

        let elapsed = now.elapsed().as_millis();
        let end_size = output.metadata().unwrap().len();
        println!(
            "[lz4 {}] Compression: {:.4}%, Time: {:.2}s",
            level,
            100.0 * end_size as f32 / start_size as f32,
            elapsed as f32 / 1000.0,
        );

        let file_bytes = std::fs::read(&out_path).unwrap();
        let decode_start = Instant::now();
        let mut decoder = lz4::Decoder::new(Cursor::new(file_bytes)).unwrap();
        let mut uncompressed_buf: Vec<u8> = vec![];
        std::io::copy(&mut decoder, &mut uncompressed_buf)
            .map_err(|e| format!("Error decoding {} as lz4: {:#?}", out_path, e))
            .unwrap();
        println!(
            "Decompress in: {:.2}s",
            decode_start.elapsed().as_millis() as f32 / 1000.0
        );
        std::fs::write(format!("lz4-in-{}.csv", level), uncompressed_buf).unwrap();
    }

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.