Add support for decompress=1 (compression of ClickHouseStatement.sendStream(..) about clickhouse-java HOT 4 CLOSED

The format of compressed blocks is described here
https://github.com/yandex/ClickHouse/blob/master/dbms/include/DB/IO/CompressedStream.h
The checksum is currently only CityHash128. I haven't found any well-known java implementation for it.
As I see checking the checksum may be turned off on server by setting a property http_native_compression_disable_checksumming_on_decompress.
So I think it is possible to support decompress=1 even without checksum.

Also as I understand it is possible to turn on gzip compression of http post content. Currently it is turned off here
https://github.com/yandex/clickhouse-jdbc/blob/master/src/main/java/ru/yandex/clickhouse/util/ClickHouseHttpClientBuilder.java#L39
It is turned off mostly because compression on selects is supported and lz4 is faster.
It is possible to turn this on at configuration of the driver, but I haven't tested this.

from clickhouse-java.

Thanks for the pointers!

After poking around a bit more, it seems it is possible to enable gzip compression for POST body (streaming) like this:

ClickHouseStatementImpl stat = (ClickHouseStatementImpl)connection.createClickHouseStatement();
stat.sendStream(new GzipCompressingEntity(new InputStreamEntity(bais, -1)), "INSERT INTO " + tableForStreaming);

This is 'almost no hackery' approach and doesn't require changes to ClickHouse JDBC driver code.

Nevertheless it would be great if you could add a better support for compression in the future (as you mentioned, lz4 is probably faster, so probably best to implement ClickHouse-style compression at the driver level).

from clickhouse-java.

Merged basic variant of support to master. Will be in the next release. At least it compresses data and works according to my tests. However, I haven't evaluated performance yet.
The property is disabled by default now and checksum is not calculated and is not checked on server side.
Some parameters like default compression buffer size and compressor option may still change.

from clickhouse-java.

Added checksumming, so this must be the last part of the issue. Decompress is still disabled by default for now.

from clickhouse-java.