martinellimarco / t2sz Goto Github PK
View Code? Open in Web Editor NEWCompress a file into a seekable zstd with special handling for .tar archives
License: GNU General Public License v3.0
Compress a file into a seekable zstd with special handling for .tar archives
License: GNU General Public License v3.0
Using the latest git commit, the -T option simply doesn't appear to work. t2sz
will use only a single thread according to htop
no matter how high I set it. This happens with -s
and -S
set to both high or low values, as well as without them set.
I am on Solus using zstd 1.5.2.
I was compressing a 250GB tar archive with -s 50K
and it oddly enough it seems to have stopped at 4GB:
ERROR: Invalid tar header. If this is not a tar archive use raw mode (-r)
Maybe the same problem as with martinellimarco/libzstd-seek#3?
Lines 74 to 75 in 84dab3b
Line 104 in 84dab3b
Currently t2sz mmaps the input file, so it has to be a physical file. This makes t2sz difficult to use in pipelines, where some stream is generated and needs to be compressed, but we don't want the uncompressed stream persisted to disk.
There's no particular reason we couldn't just read blocks into a buffer, though.
It might be nice to be able to run t2sz against non-tar-format files, eg: t2sz image.ext4 -o image.ext4.sz
. It'd be useful for mounting disk images with ratarmount, for example. My tool pixz allows this sort of thing for writing indexed xz files, and I'd love to see the same functionality for zstd.
I was wondering if t2sz supports dictionaries.
Line 15 in CMakeLists hardcodes an install prefix which does not respect the CMAKE_INSTALL_PREFIX variable.
Line 15 in 72bf850
Changing this to:
install(TARGETS t2sz)
corrects the problem and allows cmake to utilize the specified install prefix.
git clone --depth=1 --single-branch https://github.com/stedolan/jq
tar -cf jq.tar -- jq
t2sz -vfo jq.tar.zst -- jq.tar
+ <null>
# END OF BLOCK (512)
+ <null>
# END OF BLOCK (512)
munmap_chunk(): invalid pointer
[1] 164384 abort (core dumped) t2sz -vfo jq.tar.zst -- jq.tar
# OR:(sometimes): free(): invalid pointer
──────
(gdb) bt
#0 0x00007ffff7d2eef5 in raise () from /usr/lib/libc.so.6
#1 0x00007ffff7d18862 in abort () from /usr/lib/libc.so.6
#2 0x00007ffff7d70f38 in __libc_message () from /usr/lib/libc.so.6
#3 0x00007ffff7d78bea in malloc_printerr () from /usr/lib/libc.so.6
#4 0x00007ffff7d7901c in munmap_chunk () from /usr/lib/libc.so.6
#5 0x00007ffff7d7dcdb in free () from /usr/lib/libc.so.6
#6 0x00005555555552bc in main (argc=<optimized out>, argv=<optimized out>) at /home/user/.cache/aurutils/airy/t2sz-git/src/t2sz/src/t2sz.c:372
──────
(gdb) f 6
#6 0x00005555555552bc in main (argc=<optimized out>, argv=<optimized out>) at /home/user/.cache/aurutils/airy/t2sz-git/src/t2sz/src/t2sz.c:372
372 free(ctx->outFilename);
──────
(gdb) l
367 }
368 }
369
370 compressFile(ctx);
371
372 free(ctx->outFilename);
373 free(ctx);
374
375 return 0;
376 }
──────
(gdb)
==167002== Invalid free() / delete / delete[] / realloc()
==167002== at 0x483F9AB: free (vg_replace_malloc.c:538)
==167002== by 0x1092BB: main (t2sz.c:372)
==167002== Address 0x1fff000109 is on thread 1's stack
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.