mateidavid / zstr Goto Github PK

Hi,

I recently started playing with zstr in one of my projects and I noticed the library does not compile with GCC 4.8.2. I know that GCC 4.8.2 is reaaaaally old now, I think almost 10 years old, but unfortunately my users also use very old compilers...

Anyway, here is the issue (one issue at 2 places):

zstr.hpp: In member function ‘void zstr::ifstream::open(std::string, std::ios_base::openmode)’:
zstr.hpp:429:31: error: use of deleted function ‘std::basic_istream<char>& std::basic_istream<char>::operator=(const std::basic_istream<char>&)’
         std::istream::operator=(std::istream(new istreambuf(_fs.rdbuf())));

zstr.hpp: In member function ‘void zstr::ofstream::open(std::string, std::ios_base::openmode, int)’:
zstr.hpp:467:31: error: use of deleted function ‘std::basic_ostream<char>& std::basic_ostream<char>::operator=(const std::basic_ostream<char>&)’
         std::ostream::operator=(std::ostream(new ostreambuf(_fs.rdbuf(), default_buff_size, level)));

If I compile with GCC 5.4.0 everything is fine. Would there be any way to work around that?

Thank you,
Guillaume

Hi,

A friend has been trying to compile a project of mine including zstr on his Macbook (not the ARM ones but an older x86 version). Apparently, the compiler issues an error for the following line in strict_fstream.hpp

#elif ((_POSIX_C_SOURCE >= 200112L || _XOPEN_SOURCE >= 600 || defined(__APPLE__)) && ! _GNU_SOURCE) || defined(__MUSL__)

The error is error: expected value in expression and points to _GNU_SOURCE. My friend solved the issue (temporarily) by removing the ! _GNU_SOURCE. My short understanding of macros and preprocessor is that an undefined macro value initializes automatically to 0. So not sure what is the problem. Unfortunately, I do not have access to a Macbook so I cannot really test for it.

Thanks,
Guillaume

Via an extra parameter.

For analogy to zstr::ifstream that defaults to std::ifstream if input is not compressed.

In the GNU-specific strerror_r() case, line 42 here is buggy. Because the string has been swapped with "tmp", it will not contain a null character, and then it will attempt to invalidly resize to be of size std::string::npos, which is of course undesirable. (It will of course still be null terminated, but std::string::find will not find a null character within the body of the string, which is the problem).

Also, for what it's worth, 80 byte buffer here might be a bit on the small side, and may cause unnecessary truncation of errors on some OSes in some locales, at least this is my reading of searching online such as https://stackoverflow.com/questions/423248/what-size-should-i-allow-for-strerror-r#comment5466341_423301

Going to 256 or 512 might be preferable - it doesn't seem like there should be a need to be so extremely stingy with memory on modern systems.

(edit: fixed some wording, added some clarifications)

The LBANN project that uses zstr fails to compile:

In file included from /usr/local/include/zstr.hpp:15,
                 from /disk-samsung/freebsd-ports/misc/lbann/work/lbann-0.102-254-g2e3012a9a/include/lbann/data_readers/sample_list_impl.hpp:57,
                 from /disk-samsung/freebsd-ports/misc/lbann/work/lbann-0.102-254-g2e3012a9a/include/lbann/data_readers/data_reader_sample_list_impl.hpp:32,
                 from /disk-samsung/freebsd-ports/misc/lbann/work/lbann-0.102-254-g2e3012a9a/src/data_readers/data_reader_smiles.cpp:29:
/usr/local/include/strict_fstream.hpp: In function 'std::string strict_fstream::strerror()':
/usr/local/include/strict_fstream.hpp:77:26: error: invalid conversion from 'int' to 'char*' [-fpermissive]
   77 |     char * p = strerror_r(errno, &buff[0], buff.size());
      |                ~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      |                          |
      |                          int

gcc-11
FreeBSD 13.1

The various ifdefs at the beginning of strict_fstream.hpp that determine how to interact with strerror_r break for me on a Mac. Specifically, it drops into the third, GNU-specific case, but it should be using the second one. I quickly fixed my local copy by adding defined(__APPLE__) to the second case, but I'm not sure if it's the correct universal fix.

When zstr.hpp (and therefore strict_fstream.hpp) is included in multiple files, there are multiple definitions of strerror. This can easily be fixed by declaring it as static, although I do not know if link-time optimization will de-duplicate it.

I am using msys2 gcc
and I getting crashes from this
and I use Dr. memory to get some info about that error and it showed me this before crashing

Error #20: UNADDRESSABLE ACCESS beyond top of stack: reading 0x00000097d2df3700-0x00000097d2df3708 8 byte(s)
# 0 .text                                                                  [../../../gcc-10.3.0/libgcc/config/i386/cygwin.S:132]
# 1 d_demangle_callback.constprop.0                                        [C:\_\M\mingw-w64-gcc\src\build-x86_64-w64-mingw32\x86_64-w64-mingw32\libstdc++-v3\libsupc++/cp-demangle.c:6314]
# 2 __cxa_demangle                                                         [C:\_\M\mingw-w64-gcc\src\build-x86_64-w64-mingw32\x86_64-w64-mingw32\libstdc++-v3\libsupc++/cp-demangle.c:6380]
# 3 __gnu_cxx::__verbose_terminate_handler                                 [../../../../gcc-10.3.0/libstdc++-v3/libsupc++/vterminate.cc:61]
# 4 __cxxabiv1::__terminate                                                [../../../../gcc-10.3.0/libstdc++-v3/libsupc++/eh_terminate.cc:48]
# 5 std::terminate                                                         [../../../../gcc-10.3.0/libstdc++-v3/libsupc++/eh_terminate.cc:58]
# 6 __cxa_throw                                                            [../../../../gcc-10.3.0/libstdc++-v3/libsupc++/eh_throw.cc:95]
# 7 strict_fstream::detail::static_method_holder::check_open               [C:/enough_words_c_tcod/include/zstr/strict_fstream.hpp:153]
# 8 zstr::ofstream::ofstream                                               [C:/enough_words_c_tcod/include/zstr/strict_fstream.hpp:212]
# 9 deload_tile                                                            [C:/enough_words_c_tcod/src/systems/saving/dynimic_tile_loader.hpp:111]
#10 fill_roads                                                             [C:/enough_words_c_tcod/src/systems/map/huge_map_fill.hpp:133]
#11 fill_inered_roads                                                      [C:/enough_words_c_tcod/src/systems/map/huge_map_fill.hpp:186]
#12 puploate_huge_tiles                                                    [C:/enough_words_c_tcod/src/systems/map/huge_map_fill.hpp:566]
#13 Create_huge_map::init                                                  [C:/enough_words_c_tcod/src/huge_map.cpp:69]
#14 Engine::handle_events                                                  [C:/enough_words_c_tcod/src/engine.cpp:14]
#15 SDL_main                                                               [C:/enough_words_c_tcod/src/main.cpp:45]
#16 main_getcmdline                                                        [../src/main/windows/SDL_windows_main.c:126]
#17 __tmainCRTStartup                                                      [C:/_/M/mingw-w64-crt-git/src/mingw-w64/mingw-w64-crt/crt/crtexe.c:321]
#18 .l_start                                                               [C:/_/M/mingw-w64-crt-git/src/mingw-w64/mingw-w64-crt/crt/crtexe.c:202]
#19 KERNEL32.dll!BaseThreadInitThunk
Note: @0:11:20.399 in thread 26304
Note: 0x00000097d2df3700 refers to 200 byte(s) beyond the top of the stack 0x00000097d2df37c8
Note: instruction: or     $0x0000000000000000 (%rcx) -> (%rcx)

my code works correctly in the debug mode and without any optimization but when I try it with the O3 that happen
I know that isn't that helpful and I willing to share any more info if that needed, also any idea of what the cause of it is helpful

Is there a way to set / control compression parameters when opening a stream?

[Nice library! At first I tried boost, as per your suggestion. But boost's filters are not completely general; they can only write to a "sink", not to a general ostream. So that meant I couldn't write to a boost::interprocess::ovectorstream, which is what I really wanted to use as my sink. Using zstr to write to a ovectorstream works like a charm!]

Hello everyone,

I've been looking through the zstr.hpp header and I can't seem to find out how to set the compression level in zstr::ofstream?

I see zlib internally is tracking the compression level within the zstr::ostreambuf's private zstrm_p parameter. The zstr::ostreambuf object is accessible from the rdbuf() of the ofstream, but the zstrm_p of course is not.

could you share an exemple of how to use miniz with zstr on windows?

Their is no default constructor for zstr::fstream, zstr::ofstream and zstr::ifstream. The standard file streams all have default constructors, would it be possible to add this behaviour?

It appears that zstr does not open input streams in binary mode (std::ios_base::binary):

explicit ifstream(const std::string& filename, std::ios_base::openmode mode = std::ios_base::in)
    : detail::strict_fstream_holder< strict_fstream::ifstream >(filename, mode),
      std::istream(new istreambuf(_fs.rdbuf()))

Although it does for the output stream:

explicit ofstream(const std::string& filename, std::ios_base::openmode mode = std::ios_base::out)
    : detail::strict_fstream_holder< strict_fstream::ofstream >(filename, mode | std::ios_base::binary),
      std::ostream(new ostreambuf(_fs.rdbuf()))

This might be a bug? When running on Linux, I haven't found any issues, but when compiling and running on Windows, non-binary mode appears to corrupt the bytes read in, presumably because in text mode it attempts to do some Windows-specific end-of-line character conversions. Adding std::ios_base::binary to the ifstream mode fixes this.

A buffer is needed to hold the whole file after decompressing, how could I get the final file size?
It seems "seekg" and "tellg" could only get the original compressed size.

tellg(), gcount() and seekg() functions might be helpful for estimating file size.

See https://stackoverflow.com/questions/2409504/using-c-filestreams-fstream-how-can-you-determine-the-size-of-a-file

https://github.com/tmaklin/bxzstr

See what things they share and if they can be merged

The istreambuf and ostreambuf classes are given defaulted move constructors and assignment operators. However, they both inherit from std::streambuf, which is not move constructible (see https://cplusplus.github.io/LWG/issue911 for some discussion), so the default declarations don't actually accomplish anything. Clang 8 flags this via the -Wdefaulted-function-deleted warning (which is turned into an error by -Werror).

One fix would be to remove the default declarations, since they don't accomplish anything (and I can put up a PR for this if it's deemed acceptable). Another would be to add proper move constructors and assignment operators, but that's beyond the scope of my current familiarity with the library.

add useful changes from https://github.com/JGI-Bioinformatics/zstr/commits/master

so I am trying to build for android and android ndk use clang
and I am getting this error

C:\enough_words_c_tcod\include\zstr/strict_fstream.hpp:77:12: error: cannot initialize a variable of type 'char *' with
      an rvalue of type 'int'
    char * p = strerror_r(errno, &buff[0], buff.size());
           ^   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Am I missing something?
zstr.zip/tgz doesn't seem to be cpp since some if not most the logical operators have been translated to their lexical counterparts, thus rendering the lib useless for use in e.g. vcpkg:
Example from near the beginning of strict_fstream.hpp
.....
line 63 res += (not res.empty()? "|" : "");
line 72 if ((mode & std::ios_base::trunc) and not (mode & std::ios_base::out))
...
and lot's more

zstr.hpp:99 and zstr.hpp:103 use 15+32 and 12+16 as default window size. The semantics of this behavior are documented in https://zlib.net/manual.html, but I find this interface not very intuitive. It's not clear that gzip is used by default, which lead to subtle bugs in some project code.

I would appreciate if there were constants or separate parameters to make the used window size and header explicit. It would be more clear that the default value 15 + 16 refers to a 64KiB window with gzip header. Also, changing the compression header would not involve setting arbitrary numbers to the windowBits parameter.

I would propose adding an enum class type for the compression header:

enum class header {auto_detect, zlib, gzip, raw_deflate};

The streambuf classes get additional constructors which take a compression header argument and calculate the windowBits value based on the compression header and the desired window size (+32, +16 or negate window size).

I'm trying to use zstr::ifstream with a CSV parser but I'm getting:

//from the IDE (clion)
In template: call to implicitly-deleted copy constructor of 'zstr::ifstream' error occurred here 
in instantiation of member function 'csv::internals::StreamParser<zstr::ifstream>::StreamParser' requested here 
in instantiation of function template specialization 'csv::CSVReader::CSVReader<zstr::ifstream, 0>' requested here 
copy constructor of 'ifstream' is implicitly deleted because base 
class 'detail::strict_fstream_holder<strict_fstream::ifstream>' has a deleted copy constructor 
copy constructor of 'strict_fstream_holder<strict_fstream::ifstream>' is implicitly deleted because 
field '_fs' has a deleted copy constructor copy constructor of 'ifstream' is implicitly deleted because 
base class 'std::ifstream' (aka 'basic_ifstream<char>') has a deleted copy constructor 'basic_ifstream' has 
been explicitly marked deleted here

//and from the compiler
lib/zstr/src/zstr.hpp:411:7: error: use of deleted function ‘zstr::detail::strict_fstream_holder<strict_fstream::ifstream>::strict_fstream_holder(const zstr::detail::strict_fstream_holder<strict_fstream::ifstream>&)’
lib/zstr/src/zstr.hpp:400:8: note: ‘zstr::detail::strict_fstream_holder<strict_fstream::ifstream>::strict_fstream_holder(const zstr::detail::strict_fstream_holder<strict_fstream::ifstream>&)’ is implicitly deleted because the default definition would be ill-formed:
  400 | struct strict_fstream_holder
      |        ^~~~~~~~~~~~~~~~~~~~~
lib/zstr/src/zstr.hpp:400:8: error: use of deleted function ‘strict_fstream::ifstream::ifstream(const strict_fstream::ifstream&)’
lib/zstr/src/strict_fstream.hpp:177:7: error: use of deleted function ‘std::basic_ifstream<_CharT, _Traits>::basic_ifstream(const std::basic_ifstream<_CharT, _Traits>&) [with _CharT = char; _Traits = std::char_traits<char>]’
lib/zstr/src/strict_fstream.hpp:177:7: error: use of deleted function ‘std::basic_ios<_CharT, _Traits>::basic_ios(const std::basic_ios<_CharT, _Traits>&) [with _CharT = char; _Traits = std::char_traits<char>]’
  177 | class ifstream
      |       ^~~~~~~~

//this works
std::ifstream file(fp, std::ios::binary);
csv::CSVReader reader(file, csv_format);

//doesn't compile - getting the error above
zstr::ifstream file(fp, std::ios::binary);
csv::CSVReader reader(file, csv_format);

Any help appreciated.

Hi,

when i compile this library with a recent gcc (6.2.1) i get the following warning:

zstr.hpp: In destructor ‘virtual zstr::ostreambuf::~ostreambuf( ’:
zstr.hpp:271:79: warning: throw will always call terminate() [-Wterminate]
if (sync() != 0) throw Exception("~ostreambuf(): error in sink stream");

Cheers,
Georg

Hey guys,

I was wondering if it is possible to add GZIP compression functionality? The original zlib.h library contains code for GZIP compression, yet the wrapper does not.

Thanks!

How can i use zstr decompress char* or std::string data?
Could you please give me an example ?
Thanks!

This statement looks only for three specific possible zlib headers. There are many more valid zlib headers that would be missed. 61 to be precise. Look at RFC 1950 for the definition of a zlib header.