A simple benchmark to estimate the speed difference of using mutable and immutable Maps as counters.

Microbenchmarks are controversial. But we want to see their results anyway, right?

We have two tests here in this project, one using Caliper, and another wihout and such tools, using some code adapted from another test I did to compare JSON parsing libraries.

We tested three different counter update methods. We used a mutable map, an immutable map generated from a foldleft straight from the input data, and a var containing an immutable map.

Further tests also show mutable map can be about 4 times faster for integer keys.

At each test run we generated different sets of random strings. Each string had 6 characters from 'a' to 'e'. That means we generated at most 5⁶ = 15,625 different elements. We tested the methods one after the other, 11 times for each different input data size of 10k, 100k, 1M and 10M strings. In the first two cases the minimum count found at the maps was 1, while in the other the values were close to 35 and 540.

The minimum, median and maximum times from the 11 runs from each method and input sizes can be found in the tables below. We can notice the mutable map did exhibit the best times in all cases, about 40% faster than the other methods. The times from the foldLeft and the var-of-immutable methods were pretty much the same. One final interesting observation is that we can notice a somewhat linear increase in the times from 10k, 100k and 1M samples, but for 10M the time was slower. We don't know what could cause this non-linearity, we tried to use iterators for everything, and the memory consumption from the last two cases should be the same. It could be because of how memory is pre-allocated in immutable maps.

Results follow:

Caliper showed similar results, at most ~40% faster for mutable.