Add alternative compression option about leveled HOT 5 CLOSED

There's a branch - mas-i95-lz4sst

rebar ct test runs appear to be consistently quicker (but only by about o(1) %).

Running a full volume test in Riak, we can see how the sst slot_fetch time has changed between this test and master:

index=* SST01 | rex "SST timing for result (?<ResultType>[a-z_]+) is sample (?<SampleSize>[0-9]+) total (?<TotalTime>[0-9]+) and max [0-9]+" | stats sum(TotalTime) AS TotalTimeAll sum(SampleSize) as SampleSizeAll by ResultType | eval MeanTimeAll = TotalTimeAll/SampleSizeAll 

Timings are in microseconds. So using LZ4 reduces the time it takes to fetch from a slot by 45 microseconds.

45 microseconds is irrelevant in the context of GET performance, but if we're doing 5K GETs per second on a 5 node cluster this will involve 4 slot_fetches per GET - and so it amounts to about 20% of one CPU core in CPU activity. When CPU bound, this may make an overall difference.

When the SST file is being created, there appears to be a small delta with the SST file write times taking slightly longer in the test:

index=* SST08 | rex "at level (?<LedgerLevel>[0-9]{1}) with max sqn [0-9]+ with time taken (?<SSTCreateTime>[0-9]+) ms"  | stats mean(SSTCreateTime) by LedgerLevel 

Timings are in milliseconds. So the cost of creating a SST file has increased marginally (but creating a SST file. At peak in these tests though, each node is seeing < 4 SST file creations per second. So the CPU cost of this additional millisecond isn't relevant.

Tests are still running, I will post a comment with graphs on the impact of overall throughput this evening.

from leveled.

Chart below shows the difference with and without LZ4 compression of the SST (rather than using term_to_binary [compressed]). the throughput with LZ4 improves by 3% . The most notable performance improvement is a 12% reduction in 2i query time.

from leveled.

The other aspect of compression in Leveled is compression in the Journal. The Journal does not by default compress values (in the current master), it only compresses the values during compaction.

The current compression is zlib. Turning compression on on receipt (so compress immediately not after compaction), doubles the time taken by the Inker to PUT the object (1ms -> 2ms). This increase leads to a drop in throughput of 6.5%.

This drop in throughput was why the compression was disabled.

So what if LZ4 compression is used for the Journal - can this 1ms PUT overhead be reduced?

from leveled.

Switching the Journal compression to LZ4 - and actively compressing on receipt, had a negligible impact on throughput.

There is a 184 microsecond increase in the Inker PUT time, compared to disabling compression on receipt. This compares favourably to the 1ms increase from using zlib.

Although this CPU time will have an impact on throughput, it is a change with potential compensating improvements from reduced object sizes etc.

from leveled.

Based on these results - NIF support has been merged in:

#99

from leveled.