TranscodingStreams.jl

TranscodingStreams.jl is a package for transcoding data streams, which is:

fast: small overhead and specialized methods,
consistent: basic I/O operations you already know will work as you expect,
generic: support any I/O objects like files, buffers, pipes, etc., and
extensible: you can define a new codec to transcode data.

Installation

Pkg.add("TranscodingStreams")

Installing a codec package will install TranscodingStreams.jl as well, and so in general you don't need to explicitly install it.

Usage

using TranscodingStreams, CodecZlib

# Some text.
text = """
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean sollicitudin
mauris non nisi consectetur, a dapibus urna pretium. Vestibulum non posuere
erat. Donec luctus a turpis eget aliquet. Cras tristique iaculis ex, eu
malesuada sem interdum sed. Vestibulum ante ipsum primis in faucibus orci luctus
et ultrices posuere cubilia Curae; Etiam volutpat, risus nec gravida ultricies,
erat ex bibendum ipsum, sed varius ipsum ipsum vitae dui.
"""

# Streaming API.
stream = IOBuffer(text)
stream = TranscodingStream(GzipCompressor(), stream)
stream = TranscodingStream(GzipDecompressor(), stream)
for line in eachline(stream)
    println(line)
end
close(stream)

# Array API.
array = Vector{UInt8}(text)
array = transcode(GzipCompressor, array)
array = transcode(GzipDecompressor, array)
@assert text == String(array)

# Pre-allocated API
import TranscodingStreams: Buffer
compressed_buffer = Buffer(Vector{UInt8}(undef, ncodeunits(text)))
decompressed_buffer = Buffer(codeunits(text))
transcode(GzipCompressor, decompressed_buffer, compressed_buffer)
transcode(GzipDecompressor, compressed_buffer, decompressed_buffer)
@assert text == String(decompressed_buffer.data)

Each codec has an alias to its transcoding stream type for ease of use. For example, GzipCompressorStream{S} = TranscodingStream{GzipCompressor,S} where S<:IO.

Consult the docs for more details and examples.

Codec packages

TranscodingStreams.jl offers I/O interfaces to users. It also offers a protocol suite to communicate with various codecs. However, specific codecs are not included in this package except the Noop codec, which does nothing to data. The user need to install codecs as a plug-in to do something meaningful.

The following codec packages support the protocol suite:

jakobnissen / transcodingstreams.jl Goto Github PK