Profiling of CBF reading performances... about fabio HOT 6 CLOSED

the current values is (for 200 Pilatus 2M images):
Decompression byte ofset: 1.4s
MD5 sum: 0.7s
numpy.astype: 0.5s
total: 2.8s

Both astype and MD5 could be skipped when speed is really needed

from fabio.

After simple optimization on reaches 1.8s.
the byte_offset should ideally provide int32 (instead of int64 in the generic case) which would prevent the memcopy & cast

from fabio.

Hi Jerome,

Vadim's code in cryio might be useful for doing a benchmarking
comparison as it has already been optimized. It can be found at:
http://www.3lp.cx/cryio/

Back in November 2014 we saw:

In [23]: %timeit ret = cryio_cbfimage.CbfImage("run2_1_00148.cbf")
10 loops, best of 3: 29.8 ms per loop

In [25]: %timeit ret = fabio.open("run2_1_00148.cbf")
10 loops, best of 3: 150 ms per loop

I seem to remember that it was before some cif optimisations...

On 21/03/2016 17:34, Jerome Kieffer wrote:

After simple optimization on reaches 1.8s.
the byte_offset should ideally provide int32 (instead of int64 in the
generic case) which would prevent the memcopy & cast

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
https://github.com/kif/fabio/issues/52#issuecomment-199369073

from fabio.

After optimization, reading of 200 files takes 1.17s (0.93s for the decompression and 0.09s for the read itself)

In [14]: %timeit fabio.cbfimage.CbfImage().read("run2_1_00148.cbf")
10 loops, best of 3: 20.7 ms per loop

%timeit fabio.cbfimage.CbfImage().read("run2_1_00148.cbf", check_MD5=False)
100 loops, best of 3: 12.4 ms per loop

from fabio.

Hi

Vadim's code in cryio might be useful for doing a benchmarking
comparison as it has already been optimized. It can be found at:
http://www.3lp.cx/cryio/

It's http://hg.3lp.cx/cryio

Back in November 2014 we saw:

In [23]: %timeit ret = cryio_cbfimage.CbfImage("run2_1_00148.cbf")
10 loops, best of 3: 29.8 ms per loop

In [25]: %timeit ret = fabio.open("run2_1_00148.cbf")
10 loops, best of 3: 150 ms per loop

I may try to run the tests again, to compare it.

Vadim

from fabio.

On Mon, 21 Mar 2016 10:35:28 -0700
Jon Wright [email protected] wrote:

Hi

Vadim's code in cryio might be useful for doing a benchmarking
comparison as it has already been optimized. It can be found at:
http://www.3lp.cx/cryio/

To be fair, run the test on scisoft13 ...

now >90% of the time is spent in the "byte-offset decompression"
algorithm which we have checked to be at the same speed (in 2014).

Cheers,

Jerome

from fabio.