Provide a (default) wrapper for images from Eiger detector.

implement a factory + registry of all classes inheriting from fabioimage.

Dear all,

for some reason, I already have the image opened and the content in form of a string before passing it to fabio. There I write the content to a file to be able to open it again.
Is there a way to directly pass the string or a StringIO object to fabio? Up to now it wasn't obvious how to do it yet. If not.. can it be implemented with reasonable efforts?

They are similar to EDF or ADSC ... little adaptation should be needed

From: Andrew GOETZ [email protected]
To: Jonathan WRIGHT [email protected], Jerome Kieffer [email protected]
Subject: Re: Problem reading pgm file with fable
Date: Fri, 12 Sep 2014 14:00:00 +0200
Organization: ESRF
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0

I tried this and get this stack dump:

fabio.open("image0001.pgm")
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python2.7/dist-packages/fabio/openimage.py", line 113,
in openimage
obj = obj.read(filename, frame)
File "/usr/lib/python2.7/dist-packages/fabio/pnmimage.py", line 93,
in read
self._readheader(infile)
File "/usr/lib/python2.7/dist-packages/fabio/pnmimage.py", line 73,
in _readheader
m = int(self.header['MAXVAL'])
ValueError: invalid literal for int() with base 10: ''

On 12/09/2014 13:52, Jonathan WRIGHT wrote:

Hi Andy,

I get permission denied on that folder. Can you copy the image to
/tmp_14_days - otherwise running:

python -c "import fabio;
fabio.open("~goetz/workspace/data/examples/image0001.pgm")

...should give a more useful stack dump.

I still did not understand what changed with fable to make imageviewer
working? I have seen images today from the image navigator, but could
not yesterday? Strange.

Thanks!!

Jon

On 12/09/2014 13:19, Andrew GOETZ wrote:

Hi Jerome + Jon,

I have a problem reading a pgm file with fable_gui using fabio
underneath. Do you know if something has changed in fabio between
python 2.6 and 2.7 for pgm files?

The issue is described here:

http://jira.esrf.fr:8080/jira/browse/DASS-573

The file in question is :

~goetz/workspace/data/examples/image0001.pgm

Your feedback would be useful.

Andy

automatic debian packaging for debian8 (Jessie) of the development version is
currently not working. This can be fixed by squeezing the "third party
removal" patch from Fred. (see mailing list thread)

Sourceforge has become unreliable: it often breaks continuous integration as testimages are not available.

import fabio
obj = fabio.open("any.edf")
obj.header["missing"]="blah"
obj.write("any.edf")

same as silx-kit/pyFAI#307

It ends with a 404 error during the images download.

It seems to me that we should add the images in the repository in order to avoid this problem and simplify the code.
nowadays 100 Mo is not a big deal.
This is done only the first time you do the checkout

tests are missing

Apparently it is in the byte offset implementation in cython

if possible ...

for now TIffIO is imported from pymca, so I need to import all pymca during the build of fabio.
Would it be possible to move this TiffIO into fabio and make PyMCa depends on fabio.

this would avoid circular depdencies the day pymca will import fabio for x, y, z, reason.

Implies switching from lower case class name to CamelCase class names !

raxisimage.py:191: DeprecationWarning: Implicitly casting between incompatible kinds. In a future numpy release, this will raise an error. Use casting="unsafe" if this is intentional.

data[di] *= sf

import fabio
for i in range(0, 1000 ):
# Source
fnsrc = 'sample_mar_file.mar2300'
# Reading
print 'Reading {}'.format(fnsrc)
im = fabio.open(fnsrc);
raw_input('Press a key to exit')

Hi Jerome,

I have tried to use fabio to read edfs files from a network drive, but it seems that it can't them read properly. Could you advise something?

import os
import numpy as np
import re
import fabio as fb

dir_path = "Z:\\grif\\ANKA_data\\2014\\XRegioMay2014\\tomography\\Phenotyping\\Recon\\fish204\\Complete\\Corr\\fish204_proj_7529.edf"

print "os.path.exists = %d" % os.path.exists(dir_path)
print "os.path.isfile = %d" % os.path.isfile(dir_path)

img = fb.open(dir_path)
print img.header

os.path.exists = 1
os.path.isfile = 1

---------------------------------------------------------------------------
IOError                                   Traceback (most recent call last)
<ipython-input-71-be24ca875048> in <module>()
      9 print "os.path.isfile = %d" % os.path.isfile(dir_path)
     10 
---> 11 img = fb.open(dir_path)
     12 print img.header

C:\Anaconda\lib\site-packages\fabio\openimage.pyc in openimage(filename, frame)
    118     else:
    119         logger.debug("Attempting to open %s" % (filename))
--> 120         obj = _openimage(filename)
    121         logger.debug("Attempting to read frame %s from %s" % (frame, filename))
    122         obj = obj.read(obj.filename, frame)

C:\Anaconda\lib\site-packages\fabio\openimage.pyc in _openimage(filename)
    153     except IOError as error:
    154         logger.error("%s: File probably does not exist", error)
--> 155         raise error
    156     except:
    157         try:

IOError: [Errno 2] No such file or directory: '\\grif\\ANKA_data\\2014\\XRegioMay2014\\tomography\\Phenotyping\\Recon\\fish204\\Complete\\Corr\\fish204_proj_7529.edf'

Many error on such computers ...
used mainly for the Debian continuous integration system

Hello, when I download the pacakge from pip, I get the pylint directory.

Is it possible to remove this directory from the sdource distribution.

Thanks

Hi,

thanks very much for your fabio library. it is really great.

I recently came across OXD files with the TY5 compression that fabio cannot read.
Would it be possible to add the support for this compression ?

I am not an expert in OXD files so I can't really help very much unfortunately,

thanks a lot,
marco

ps. I can send one file if needed

Hello Jerome,

I get this error message

Traceback (most recent call last):
File "/usr/bin/fabio_viewer", line 189, in open_h5_data_series
entry = nxs.get_entries()[0]
File "/usr/lib/python2.7/dist-packages/fabio/nexus.py", line 231, in get_entries
self.h5[grp].attrs["NX_class"] == "NXentry")]
File "/usr/lib/python2.7/dist-packages/fabio/nexus.py", line 143, in from_isotime
return time.mktime(time.strptime(base, "%Y-%m-%dT%H:%M:%S")) + tz
File "/usr/lib/python2.7/_strptime.py", line 467, in _strptime_time
return _strptime(data_string, format)[0]
File "/usr/lib/python2.7/_strptime.py", line 325, in _strptime
(data_string, format))
ValueError: time data "['2016-03-23T12:01:" does not match format '%Y-%m-%dT%H:%M:%S'

Not sure if this is desired behaviour, but in my experience, most manufacturer file formats consist of a header at the beginning of the file followed by a binary block.
Therefore, when provided with image dimensions, it would be convenient to have fabIO.binaryimage read a datablock of the required size from the end of the file rather than the beginning of the file.
To prevent backward compatibility problems, the "seek from end of file" option could be enabled by setting "offset=-1".

Hei,

i have a user of Dioptas who has a lot of .f2d files. Do you plan to implement reading of the "Fit2d" file format into fabio?

You already support fit2dmaskimage, and fit2dspreadsheetimage.

I could not find any specifications online for the .f2d file format, but since you are close to hammersley, maybe you can get it from him.

Best wishes,
Clemens

Hello, when preparing the debian package, I remove the third_party to avoid embeding code.
But then during the build, the process stoped with

copying fabio-src/pnmimage.py -> /tmp/buildd/python-fabio-0.2.1+dfsg1/.pybuild/pythonX.Y_2.7/build/fabio
error: package directory 'third_party' does not exist
E: pybuild pybuild:256: build: plugin distutils failed with: exit code=1: /usr/bin/python setup.py build
dh_auto_build: pybuild --build -i python{version} -p 2.7 --dir . returned exit code 13
debian/rules:22: recipe for target 'override_dh_auto_build-indep' failed

Is it possible to fix the seup.py to check for the third_party directory existance ?
Then I will add the right dependencies for argparses,six etc...

thanks

Fred

/compression.py:310: DeprecationWarning: numpy boolean subtract (the binary - operator) is deprecated, use the bitwise_xor (the ^ operator) or the logical_xor function instead.

exception16 = (adiff >= 127) - exception32 # 2**7-1)

On reading BKGINIT.cbf or any other XDS generated file fabio gives an error:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-63-543cb060b3e7> in <module>()
      1 import fabio
----> 2 f=fabio.open("/home/arkadiy/work/ID23_042013/Urea12/xds/ABSORP.cbf")

/usr/local/lib/python2.7/dist-packages/fabio/openimage.pyc in openimage(filename, frame)
    111         obj = _openimage(filename)
    112         logger.debug("Attempting to read frame %s from %s" % (frame, filename))
--> 113         obj = obj.read(filename, frame)
    114     return obj
    115 

/usr/local/lib/python2.7/dist-packages/fabio/cbfimage.pyc in read(self, fname, frame)
    205         binary_data = self.cbs[self.start_binary + len(STARTER):]
    206         logger.debug("CBS type %s len %s" % (type(self.cbs), len(self.cbs)))
--> 207         ref = numpy.string_(self.header["Content-MD5"])
    208         obt = md5sum(binary_data)
    209         if ref != obt:

KeyError: 'Content-MD5'

Which is most probably caused by the fact that XDS does not include MD5 sum of the binary file in it.

The example of XDS file can be found here:
https://drive.google.com/file/d/0B0M8iPoXKKOrNWRGaGFLSm9FVFU/view?usp=sharing

doc/source/conf.py: No copyright GENERATED FILE
fabio-src/raxisimage.py: No copyright UNKNOWN
fabio-src/_version.py: No copyright UNKNOWN
fabio-src/openimage.py: No copyright UNKNOWN
fabio-src/marccdimage.py: No copyright UNKNOWN
fabio-src/init.py: No copyright UNKNOWN
fabio-src/pilatusimage.py: No copyright UNKNOWN
fabio-src/fit2dspreadsheetimage.py: No copyright UNKNOWN
fabio-src/pixiimage.py: No copyright UNKNOWN
fabio-src/edfimage.py: No copyright UNKNOWN
fabio-src/adscimage.py: No copyright UNKNOWN
fabio-src/GEimage_old.py: No copyright UNKNOWN
fabio-src/OXDimage.py: No copyright UNKNOWN
fabio-src/mar345image.py: No copyright UNKNOWN
fabio-src/xsdimage.py: No copyright UNKNOWN
fabio-src/directories.py: No copyright UNKNOWN
fabio-src/converters.py: No copyright UNKNOWN
fabio-src/nexus.py: GPL (v3 or later)
fabio-src/mrcimage.py: No copyright UNKNOWN
fabio-src/GEimage.py: No copyright UNKNOWN
fabio-src/templateimage.py: No copyright UNKNOWN
fabio-src/readbytestream.py: No copyright UNKNOWN
fabio-src/brukerimage.py: No copyright UNKNOWN
fabio-src/tifimage.py: UNKNOWN
fabio-src/fabioimage.py: No copyright UNKNOWN
fabio-src/bruker100image.py: No copyright UNKNOWN
fabio-src/cbfimage.py: No copyright GENERATED FILE
fabio-src/compression.py: No copyright UNKNOWN
fabio-src/hdf5image.py: No copyright UNKNOWN
fabio-src/dm3image.py: No copyright UNKNOWN
fabio-src/kcdimage.py: No copyright UNKNOWN
fabio-src/fit2dmaskimage.py: No copyright UNKNOWN
fabio-src/file_series.py: No copyright UNKNOWN
fabio-src/datIO.py: No copyright UNKNOWN
fabio-src/fabioutils.py: No copyright UNKNOWN
fabio-src/HiPiCimage.py: No copyright UNKNOWN
fabio-src/binaryimage.py: No copyright UNKNOWN
fabio-src/pnmimage.py: No copyright UNKNOWN
setup.py: No copyright GENERATED FILE
src/stdint.h: BSD (2 clause)
src/columnfile.c: No copyright UNKNOWN
src/ccp4_pack.c: LGPL (v3 or later)
src/mar345_IO.pyx: No copyright UNKNOWN
src/columnfile.h: No copyright UNKNOWN
src/ccp4_pack.h: LGPL (v3 or later)
src/cf_io.pyx: No copyright UNKNOWN
src/cf_iomodule.c: No copyright UNKNOWN
src/byte_offset.pyx: No copyright UNKNOWN
src/_cif.pyx: No copyright UNKNOWN
test/testtifimage.py: No copyright UNKNOWN
test/init.py: GPL (v3 or later)
test/testadscimage.py: No copyright UNKNOWN
test/test_flat_binary.py: No copyright UNKNOWN
test/testGEimage.py: No copyright UNKNOWN
test/benchheader.py: No copyright UNKNOWN
test/testcbfimage.py: No copyright UNKNOWN
test/testopenheader.py: No copyright UNKNOWN
test/testfilenames.py: No copyright UNKNOWN
test/utilstest.py: UNKNOWN
test/testkcdimage.py: No copyright UNKNOWN
test/testOXDimage.py: No copyright UNKNOWN
test/testbruker100image.py: No copyright UNKNOWN
test/testedfimage.py: No copyright UNKNOWN
test/testfit2dmaskimage.py: No copyright UNKNOWN
test/profile_compressed.py: No copyright UNKNOWN
test/testcompression.py: No copyright UNKNOWN
test/testraxisimage.py: No copyright UNKNOWN
test/test_filename_steps.py: No copyright UNKNOWN
test/testmccdimage.py: No copyright UNKNOWN
test/profile/benchheader.py: No copyright UNKNOWN
test/profile/profile_compressed.py: No copyright UNKNOWN
test/test_all_images.py: No copyright UNKNOWN
test/testfabioimage.py: No copyright UNKNOWN
test/testbrukerimage.py: No copyright UNKNOWN
test/testheadernotsingleton.py: No copyright UNKNOWN
test/testpnmimage.py: No copyright UNKNOWN
test/test_all.py: No copyright UNKNOWN
test/bug16_ppc64le.py: No copyright UNKNOWN
test/test_file_series.py: No copyright UNKNOWN
test/testopenimage.py: No copyright UNKNOWN
test/testXSDimage.py: No copyright UNKNOWN
test/testmar345image.py: No copyright UNKNOWN
third_party/six.py: MIT/X11 (BSD like)
third_party/init.py: No copyright UNKNOWN
third_party/argparse.py: Apache (v2.0)
third_party/gzip.py: No copyright UNKNOWN
third_party/TiffIO.py: MIT/X11 (BSD like)

Modify parameters ?
Traceback (most recent call last):
File "/usr/bin/pyFAI-calib", line 65, in
c.postProcess()
File "/usr/lib/pymodules/python2.6/pyFAI/calibration.py", line 403, in postProcess
polarization_factor=self.polarization_factor)[0]
File "/usr/lib/pymodules/python2.6/pyFAI/azimuthalIntegrator.py", line 1705, in xrpd2_splitBBox
self.save2D(filename, I, bins2Th, binsChi)
File "/usr/lib/pymodules/python2.6/pyFAI/azimuthalIntegrator.py", line 2754, in save2D
img.write(filename)
File "/usr/lib/pymodules/python2.6/fabio/edfimage.py", line 658, in write
outfile = self._open(fname, mode="wb")
File "/usr/lib/pymodules/python2.6/fabio/fabioimage.py", line 376, in _open
self.filenumber = fabioutils.extract_filenumber(fname)
File "/usr/lib/pymodules/python2.6/fabio/fabioutils.py", line 249, in extract_filenumber
obj = deconstruct_filename(name)
File "/usr/lib/pymodules/python2.6/fabio/fabioutils.py", line 201, in deconstruct_filename
num = int(numstring)
ValueError: invalid literal for int() with base 10: ''

When trying to open an image using FabIO on Python 3 an error message is presented: 'str' does not support the buffer interface.

In [1]: fname = "lysb_5mg-1.90s_SAXS"

In [2]: import fabio

In [3]: fabio.open(name)

The file can be downloaded from the following link: https://db.tt/FInF99Fz

% fabio_viewer
Traceback (most recent call last):
File "/usr/bin/fabio_viewer", line 105, in open_data_series
series = [str(f) for f in list(series)]
TypeError: 'bool' object is not iterable

Hi Kif,

Thanks for the great fabio project. Recently I got confusing of converting these frames collected by an ADSC detector to Bruker files. It is really a surprise when I found the fabio project.
It shows the .img file could be read by fabio and .sfrm file could be write by fabio. However, with the new fabio_viewer, it do have a choice to save as .sfrm file. However, this sfrm file can not be read by bruker apex II, can you help me about this. also, the automatic format recognition procedure seems didn't work correctly. Please help me

Requested by ID31 & ID15.

Hi,
I installed the latest version of FabIO from github master and the tests passed.
The issue is that I'm trying to use FabIO to open a TIFF image but the following error is shown:

In [1]: import fabio

In [2]: fabio.open("Feb09-bright-00.300s_WAXS")

WARNING:tifimage:Unable to read Feb09-bright-00.300s_WAXS with TiffIO due to could not broadcast input array from shape (521,1042) into shape (1042,1042), trying PIL
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
<ipython-input-2-ce38474c489e> in <module>()
----> 1 fabio.open("Feb09-bright-00.300s_WAXS")

/Users/hugoslepicka/anaconda/envs/pyXS/lib/python3.4/site-packages/fabio-0.3.0b3-py3.4-macosx-10.5-x86_64.egg/fabio/openimage.py in openimage(filename, frame)
    144         obj = _openimage(filename)
    145         logger.debug("Attempting to read frame %s from %s" % (frame, filename))
--> 146         obj = obj.read(obj.filename, frame)
    147     return obj
    148

/Users/hugoslepicka/anaconda/envs/pyXS/lib/python3.4/site-packages/fabio-0.3.0b3-py3.4-macosx-10.5-x86_64.egg/fabio/tifimage.py in read(self, fname, frame)
    188                         self.data = numpy.fromstring(self.pilimage.tostring(), PIL_TO_NUMPY[self.pilimage.mode])
    189                     else:  # probably RGB or RGBA images: rely on PIL to convert it to greyscale float.
--> 190                         self.data = numpy.fromstring(self.pilimage.convert("F").tostring(), numpy.float32)
    191                     self.data.shape = (self.dim2, self.dim1)
    192             else:

/Users/hugoslepicka/anaconda/envs/pyXS/lib/python3.4/site-packages/PIL/Image.py in tostring(self, *args, **kw)
    684     def tostring(self, *args, **kw):
    685         raise Exception("tostring() has been removed. " +
--> 686                         "Please call tobytes() instead.")
    687
    688     def tobitmap(self, name="image"):

Exception: tostring() has been removed. Please call tobytes() instead.

I can open the image with the following code:

In [1]: from PIL import Image

In [2]: im = Image.open("Feb09-bright-00.300s_WAXS")

In [3]: import numpy as np

In [4]: import matplotlib.pyplot as plt

In [5]: data = np.asarray(im).copy()

In [6]: plt.imshow(data)

In [7]: plt.show()

Packages installed:

python 3.4.3
fabio 0.3.0b3
pillow 3.0.0

Thanks,
Hugo

There is a memory management issue with large multiframe EDF images.

Offering an iterator over them (with explicit destruction of data) would solve it.

The filenames stored in the header may have the drive prefix ("c:") or not which makes some test fail under windows.
Minor issue.

Systematically use a tearDown method which clean up the created files !!!

Hi, Man,
I have a question in the reading the tif image generated by the PILATUS detector. In the user manual of the pilatus detector, they said that the generated tif is 32-bit unsigned integers. If that ,therefore, we should use the uint32 when we read the tid data into an array. However, in the Fabio, I went to the tifimage.read and found it doesn't use the uint32 but the int32. I can't understand why we should use the int32 not the uint32. In practice, I tried to use the uint32 to convert the data but found the data is not correct. In principle, I infer that the int32 should be correct. Does this mean the image format description in the user manual is not correct? Thanks a lot.

Mainly related to CIF parsing in Python ...
Further optimization: calculate md5 together with decoding of the binary blob.

thanks to Jon for reporting it... with the patch and the test-case.

Hei Jérôme,

I had some users who also collect diffraction data with Princeton Instrument Detectors. Since I already wrote a small library to open the version 2 and version 3 .spe formats. I know how to read the fileformat. Would you be interested that I add the file format it to the fabio library?

Best wishes,
Clemens

• dim1, dim2, bpp, ... and many other features could be derived from self.data using propoerties
• use "ordered dict" for storing header to avoid using together with header_keys list

Example of file:
http://vincefn.net/psic_nano_20150220bis_fast_00010_004_0134.edf.gz

Issue: wrong file size read.

Hei there,

a user of Dioptas just reported a bug when loading edf files from a MAR3450 detector from the PSICHE beamline at SOLEIL. Reading the file with fabio gives:

WARNING:edfimage:Unable to read header size in header_length ;
ByteOrder
ERROR:edfimage:Unable to locate start of the binary section
WARNING:edfimage:no opening {. Corrupt header of EDF file /home/clemens/Downloads/test.edf

And results in a shifted image:

Fit2D is able to read the file without problems and they told me that it only doesn't work for the highest resolution files. The example file can be downloaded from:

https://uni-koeln.sciebo.de/index.php/s/K5lswUJNVqnMKKT