OXD images (with TY5 compression) about fabio HOT 12 CLOSED

On Wed, 11 Mar 2015 04:09:31 -0700
marcocamma [email protected] wrote:

I recently came across OXD files with the TY5 compression that fabio cannot read.
Would it be possible to add the support for this compression ?

I never heard of it but it could be possible.

I am not an expert in OXD files so I can't really help very much unfortunately,

Maybe we can ask Agilent for information ... Sometimes manufacturers answer positively: Bruker, ImXpad and Rayonix did it, why not Agilent ?

ps. I can send one file if needed

That would definitely help. If you could also send the same data under another more common format (tiff, ...) it would help to set-up tests.

thanks

Jérôme Kieffer

from fabio.

thanks very much !

attached please find the same image as recorded (_TY5) and converted using
crysalis without compression (_uncompressed),

regards,
marco

marco cammarata

On 12 March 2015 at 15:43, Jerome Kieffer [email protected] wrote:

On Wed, 11 Mar 2015 04:09:31 -0700
marcocamma [email protected] wrote:

I recently came across OXD files with the TY5 compression that fabio
cannot read.
Would it be possible to add the support for this compression ?

I never heard of it but it could be possible.

I am not an expert in OXD files so I can't really help very much
unfortunately,

Maybe we can ask Agilent for information ... Sometimes manufacturers
answer positively: Bruker, ImXpad and Rayonix did it, why not Agilent ?

ps. I can send one file if needed

That would definitely help. If you could also send the same data under
another more common format (tiff, ...) it would help to set-up tests.

thanks

Jérôme Kieffer

—
Reply to this email directly or view it on GitHub
https://github.com/kif/fabio/issues/19#issuecomment-78493111.

from fabio.

On Fri, 13 Mar 2015 05:43:27 -0700
marcocamma [email protected] wrote:

thanks very much !

attached please find the same image as recorded (_TY5) and converted using
crysalis without compression (_uncompressed),

Hi,

the attachement got dropped :(
could you send this directly to my professional email address

Thanks.

J�r�me Kieffer [email protected]

from fabio.

Hello,
i also came across some OXD files with the TY5 compression.
I think there is some additional information in the bitstream of the file, which produces artifacts in the resulting picture.
If it helps i can provide some data and pictures.
kind regards
Philip

from fabio.

On Thu, 09 Jul 2015 05:57:12 -0700
PhHarder [email protected] wrote:

Hello,
i also came across some OXD files with the TY5 compression.
I think there is some additional information in the bitstream of the file, which produces artifacts in the resulting picture.
If it helps i can provide some data and pictures.

The big problem, to my understanding, is the fact that TY5 files we got
up to now are raw (and wrapped) and uncompressed files have been unwrapped (i.e.
corrected for geometric distortion).
As we are searching bit-patterns to perform the reverse-engineering, if
the two binary blobs are not exactly the same it is not worth spending time on it.
This is my point of view, if somebody finds it funny, I am ready to
help.

According to the ratio of data-compression I suspect a byte-offset type
of algorithm, but I have no other evidence.

Cheers,

Jerome.
PS: For now, I will start recommending not to purchase any
Oxford/Agilent/Rigaku equipment: all other manufacturer have been much
more open-minded, providing at least specification if not the source code !!!

from fabio.

On 09/07/2015 21:14, Jerome Kieffer wrote:

On Thu, 09 Jul 2015 05:57:12 -0700
PhHarder [email protected] wrote:

Hello,
i also came across some OXD files with the TY5 compression.
I think there is some additional information in the bitstream of the
file, which produces artifacts in the resulting picture.
If it helps i can provide some data and pictures.

The big problem, to my understanding, is the fact that TY5 files we got
up to now are raw (and wrapped) and uncompressed files have been
unwrapped (i.e.
corrected for geometric distortion).
As we are searching bit-patterns to perform the reverse-engineering, if
the two binary blobs are not exactly the same it is not worth spending
time on it.
This is my point of view, if somebody finds it funny, I am ready to
help.

According to the ratio of data-compression I suspect a byte-offset type
of algorithm, but I have no other evidence.

Cheers,

Jerome.
PS: For now, I will start recommending not to purchase any
Oxford/Agilent/Rigaku equipment: all other manufacturer have been much
more open-minded, providing at least specification if not the source
code !!!

Semi-related, there are notes on writing esperanto and code from Pascal
Parois here:

http://blog.debroglie.net/2014/07/09/compression-and-crystallograpic-2d-images/
https://redmine.debroglie.net/projects/debroglie/repository/entry/Oxford/trunk/diamond2crysalis/bitfield.F90

I have no idea if it is like "TY5"? My understanding is that if you have
data in TY5 format then you got it from an Oxford detector and it is
their "raw" format. It seems they prefer their customers to correct the
data using their software and write it out in another format instead of
reading it directly. This sort of makes sense.

For anyone thinking of using "esperanto", the open version is 32-bit
integers and the compressed version is the secret algorithm which Pascal
has worked out. This means frames of 8MB can become 16MB or 3MB,
depending if the compression is used.

All that aside, if there is a new test case it would be useful to add it
to the testimages collection here:

http://sourceforge.net/projects/fable/files/test_images/

There is always a chance someone else will come along and fix the code
in fabio if they are also an Oxford customer. I am cross posting this on
the fable list in case someone there knows more...

Best,

Jon

from fabio.

Hello again,
thanks for the tips. Here is another link:
http://www.iucr.org/__data/iucr/cif/software/cbflib/CBFlib_0.7.9/doc/CBFlib.html#3.3.3
Chapter 3.3.3 describes the method of byte-offset compression.
I think this is what the TY5 compression does.
I wrote a little python script to open a file. you can try it here:
http://pastebin.com/W866xUWJ
Cheers,
Philip

from fabio.

On Tue, 11 Aug 2015 02:16:44 -0700
PhHarder [email protected] wrote:

Hello again,
thanks for the tips. Here is another link:
http://www.iucr.org/__data/iucr/cif/software/cbflib/CBFlib_0.7.9/doc/CBFlib.html#3.3.3
Chapter 3.3.3 describes the method of byte-offset compression.
I think this is what the TY5 compression does.
I wrote a little python script to open a file. you can try it here:
http://pastebin.com/W866xUWJ

Thanks a lot, it definitely helps

Jérôme Kieffer
tel +33 476 882 445

from fabio.

Philip,
Are you sure of your formula (you used x255 instead of x256 ?)

Has anybody an image with 32-bit exception ?

from fabio.

I am not 100% sure about the formula.
I tried different things and this formula produced nice looking images.
Using x256 instead of x255 also works and might be better. (makes more sense)
I have some images that have the special case i can not explain, but I am not sure if this special case is a 32-bit exception.
What would be the most elegant way to make them public here?

from fabio.

On Fri, 14 Aug 2015 06:39:49 -0700
PhHarder [email protected] wrote:

I am not 100% sure about the formula.
I tried different things and this formula produced nice looking images.
Using x256 instead of x255 also works and might be better. (makes more sense)
I have some images that have the special case i can not explain, but I am not sure if this special case is a 32-bit exception.
What would be the most elegant way to make them public here?

Send them by email to me (jerome . kieffer .at. esrf . fr ... )
I will add them on the pool of test images if you agree. The testimages
may become part of Debian soon for autotomatic testing.

Thanks a lot.

Jerome

from fabio.

without test-images it is hard to write IO tests. Close issue for now.

from fabio.