question: how to use with gzipped file about csv.jl HOT 4 CLOSED

No, currently GZip streams are not supported. They were at one point, but they ended up being a total performance trap (like 1000x slower than just unzipping the file beforehand). So for now, the recommended approach is to unzip your file beforehand then parse it.

To ultimately support GZip, we need to do an overhaul of the GZip/Libz packages to combine the best of each. Currently, GZip sucks for performance, but supports file-based operations, while Libz is fast, but only supports stream-based operations. It's on my TODO list.

from csv.jl.

Apparently https://github.com/BioJulia/Libz.jl addresses (or plans to address) the same problems (but still gives the same error message). Perhaps it would be easier to make CSV work with that?

from csv.jl.

Unfortunately decompression is cumbersome, the multiple CSV files are about 1GB each and gzip -9 gives me a factor of 10-15. I have enough space, but the server is HDD-based so everything is so much faster with gzipped files.

from csv.jl.

Reading gzipped files is now supported via Libz.jl package, like

julia> f = ZlibInflateInputStream(open(joinpath(dir, "test_basic.csv.gz")))
BufferedStreams.BufferedInputStream{Libz.Source{:inflate,BufferedStreams.BufferedInputStream{IOStream}}}(<8.0 KiB buffer, 0% filled>)

julia> df = CSV.read(f)
3×3 DataFrames.DataFrame
│ Row │ col1 │ col2 │ col3 │
├─────┼──────┼──────┼──────┤
│ 1   │ 1    │ 2    │ 3    │
│ 2   │ 4    │ 5    │ 6    │
│ 3   │ 7    │ 8    │ 9    │

from csv.jl.