ccpem / mrcfile Goto Github PK

A Python implementation of the MRC2014 file format

Home Page: http://www.ccpem.ac.uk/mrc_format/mrc2014.php

License: BSD 3-Clause "New" or "Revised" License

Python 100.00%

mrcfile's Introduction

mrcfile.py

mrcfile is a Python implementation of the MRC2014 file format, which is used in structural biology to store image and volume data.

It allows MRC files to be created and opened easily using a very simple API, which exposes the file's header and data as numpy arrays. The code runs in Python 2 and 3 and is fully unit-tested.

This library aims to allow users and developers to read and write standard-compliant MRC files in Python as easily as possible, and with no dependencies on any compiled libraries except numpy. You can use it interactively to inspect files, correct headers and so on, or in scripts and larger software packages to provide basic MRC file I/O functions.

Key Features

Clean, simple API for access to MRC files
Easy to install and use
Validation of files according to the MRC2014 format
Seamless support for gzip and bzip2 files
Memory-mapped file option for fast random access to very large files
Asynchronous opening option for background loading of multiple files
Runs in Python 2 & 3, on Linux, Mac OS X and Windows

Installation

The mrcfile library is available from the Python package index:

pip install mrcfile

Or from conda-forge:

conda install --channel conda-forge mrcfile

It is also included in the ccpem-python environment in the CCP-EM software suite.

The source code (including the full test suite) can be found on GitHub.

Basic usage

The easiest way to open a file is with the mrcfile.open and mrcfile.new functions. These return an MrcFile object which represents an MRC file on disk.

To open an MRC file and read a slice of data:

>>> import mrcfile
>>> with mrcfile.open('tests/test_data/EMD-3197.map') as mrc:
...     mrc.data[10,10]
...
array([ 2.58179283,  3.1406002 ,  3.64495397,  3.63812137,  3.61837363,
        4.0115056 ,  3.66981959,  2.07317996,  0.1251585 , -0.87975615,
        0.12517013,  2.07319379,  3.66982722,  4.0115037 ,  3.61837196,
        3.6381247 ,  3.64495087,  3.14059472,  2.58178973,  1.92690361], dtype=float32)

To create a new file with a 2D data array, and change some values:

>>> array = np.zeros((5, 5), dtype=np.int8)
>>> with mrcfile.new('tmp.mrc') as mrc:
...     mrc.set_data(array)
...     mrc.data[1:4,1:4] = 10
...     mrc.data
...
array([[ 0,  0,  0,  0,  0],
       [ 0, 10, 10, 10,  0],
       [ 0, 10, 10, 10,  0],
       [ 0, 10, 10, 10,  0],
       [ 0,  0,  0,  0,  0]], dtype=int8)

The data will be saved to disk when the file is closed, either automatically at the end of the with block (like a normal Python file object) or manually by calling close(). You can also call flush() to write any changes to disk and keep the file open.

To validate an MRC file:

>>> mrcfile.validate('tests/test_data/EMD-3197.map')
File does not declare MRC format version 20140 or 20141: nversion = 0
False

>>> mrcfile.validate('tmp.mrc')
True

Documentation

Full documentation is available on Read the Docs.

Citing mrcfile

If you find mrcfile useful in your work, please cite:

Burnley T, Palmer C & Winn M (2017) Recent developments in the CCP-EM software suite. Acta Cryst. D73:469--477. doi: 10.1107/S2059798317007859

Contributing

Please use the GitHub issue tracker for bug reports and feature requests, or email CCP-EM.

Code contributions are also welcome, please submit pull requests to the GitHub repository.

To run the test suite, go to the top-level project directory (which contains the mrcfile and tests packages) and run python -m unittest tests. (Or, if you have tox installed, run tox.)

Licence

The project is released under the BSD licence.

mrcfile's People

Contributors

Stargazers

Watchers

mrcfile's Issues

Errors when running against numpy 1.16

The numpy 1.16.0 release fixes fromstring() issue that was exploited in the code.
numpy.rec.fromstring():
Arrays based off readonly buffers cannot be set writeable
We now disallow setting the writeable flag True on arrays created from fromstring(readonly-buffer).

from:
https://github.com/numpy/numpy/blob/master/doc/release/1.16.0-notes.rst

Option to calculate header.rms in float32?

I'm trying to optimize a python module that writes some files using mrcfile, and frequent mrc output hits a strange limit.

mrcfile/mrcfile/mrcobject.py

Line 562 in 87f721d

self.header.rms = np.float32(self.data.std(dtype=np.float64))

This line takes 80% of the time in setting data in a new mrc. I appreciate that the rms field needs to be set, but casting this to double just to prevent overflow is a bit much, no? If the map shows overflow in calculating the rms, the map values are probably crazy, unless I'm missing something obvious.

How does it fit in the scheme of things to either omit rms and set it to NaN, or not cast to f64 and raise an erro (or exception with a f64 fallback)?

MRC load problem

Dear all,

when I try to load an mrc file created with SerialEM (independent of fiel type, examples attached), I get the following error:

Invalid machine stamp: 0x44 0x00 0x00 0x00
Error in header labels: nlabl is 2 but 10 labels contain text

mrcfile.version gives me
<module 'mrcfile.version' from 'xxx/anaconda2/lib/python2.7/site-packages/mrcfile/version.pyc'>

and is not callable as a function

How can I figure out whether my installation is functional?

Thanks,
Martin

serialem_mrc.zip

Correction of the usage guidelines

There is a mistake in the README.rst
mrc.set_data(np.zeros((5, 5), dtype=np.int8))
This doesn't work. From one side, the data type is better to be float32 always (you don't want to confuse people right?). From the other side, dtype is not defined at all. I would write that as:

# Saving 2D or 3D matrix in mrc format:
I = np.zeros([5,5]) # your 2D or 3D matrix
out = mrcfile.new('FileName.mrc')
out.set_data(np.float32(I))
out.close()

Thanks for sharing your library.

Mohamad

Complex valued data in mrcfile gives warnings.

Hi Colin

I thought I would add a github ticket to cover what we discussed earlier.

When I save a complex valued mrc file I get the following warnings.

/home/upc86896/Software/env/dev/lib/python3.9/site-packages/numpy/core/records.py:516: ComplexWarning: Casting complex values to real discards the imaginary part
  return self.setfield(val, *res)
/home/upc86896/Software/env/dev/lib/python3.9/site-packages/numpy/core/_methods.py:178: ComplexWarning: Casting complex values to real discards the imaginary part
  ret = umr_sum(arr, axis, dtype, out, keepdims, where=where)
/home/upc86896/Software/env/dev/lib/python3.9/site-packages/numpy/core/_methods.py:211: ComplexWarning: Casting complex values to real discards the imaginary part
  arrmean = umr_sum(arr, axis, dtype, keepdims=True, where=where)

What metadata to store is obviously poorly defined but perhaps we can set these values to NaN in the header.

OverflowError: Python integer out of bounds for int32, when reading large file

I am using mrcfile version 1.5.0 to open a tomogram file with 1022x1440x500 pixels and it gave me the following error

>>> import mrcfile
>>> with mrcfile.open("ts_621_bin4_rec_trimed.mrc", permissive=True) as mrc:
...     data = mrc.data
... 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/cii/anaconda3/envs/cryocat/lib/python3.9/site-packages/mrcfile/load_functions.py", line 139, in open
    return NewMrc(name, mode=mode, permissive=permissive,
  File "/home/cii/anaconda3/envs/cryocat/lib/python3.9/site-packages/mrcfile/mrcfile.py", line 115, in __init__
    self._read(header_only)
  File "/home/cii/anaconda3/envs/cryocat/lib/python3.9/site-packages/mrcfile/mrcfile.py", line 131, in _read
    super(MrcFile, self)._read(header_only)
  File "/home/cii/anaconda3/envs/cryocat/lib/python3.9/site-packages/mrcfile/mrcinterpreter.py", line 173, in _read
    self._read_data()
  File "/home/cii/anaconda3/envs/cryocat/lib/python3.9/site-packages/mrcfile/mrcfile.py", line 137, in _read_data
    remaining_file_size = file_size - header_size
OverflowError: Python integer 2943425024 out of bounds for int32

This is an out of bound problem of np.int32 type and I fixed it by replacing header_size with int(header_size)

mrcfile/mrcfile/mrcfile.py

Line 137 in 28d72bf

remaining_file_size = file_size - header_size

imod file with unrecognized machine stamp

Hi,

Using mrcfile==1.4.3 and attempting to read a file exported from imod, i get the following warning:

with mrcfile.open('21.mrc', 'r', header_only=True, permissive=True) as mrc:
     print(mrc.header)

/home/user/miniconda3/envs/blik/lib/python3.9/site-packages/mrcfile/mrcinterpreter.py:219: 
RuntimeWarning: Unrecognised machine stamp: 0x44 0x20 0x20 0x20
  warnings.warn(str(err), RuntimeWarning)
(1024, 1024, 512, 2, 0, 0, 0, 1024, 1024, 512, (1024., 1024., 512.), (90., 90., 90.), 1, 2, 3, -995.9688, 480.2282, -2.2351742e-07,
 0, 0, b'\x00\x00\x00\x00\x00\x00\x00\x00', b'', 0, b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00
\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00
\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00
\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', (0., 0., 0.), b'MAP ', [68, 32, 32, 32], -1., 0, [b'', b'', b'', b'', b'', b'', b'', b'', 
b'', b''])

Without permissive=True this would lead to an error.

The file is from a collaborator, so I don't know whether some export options are responsible for this behaviour. If that's of any interest to the project, I can investigate!

edit: I must warn that I never worked with mrc files before, so I might just not be aware of obvious things to do when saving files in imod. :)

Issue reading files with an FEI2 extended header including unused space

According to the FEI extended header specification, the extended header may have unused space after the NZ sequential metadata blocks. I have come across a file like this that was apparently produced by Velox, where it looks like this space is used as padding to bring the total size of the extended header, NSYMBT, up to the nearest multiple of 512 bytes.

At the moment, such files cannot be read by mrcfile, because an attempt is made to coerce the full NSYMBT-long bytearray into an array of elements with data type of each metadata block:

mrcfile/mrcfile/mrcinterpreter.py

Line 292 in fdce0b5

self._extended_header.dtype = dtype

The specification document states:

Image reading and processing software can use the Metadata Size value from the first Metadata Block to index the blocks for the other frames in the MRC file.

I'm happy to try to (eventually!) put in a PR to add that behaviour, noting this warning, as discussed by email:

We might need a new attribute or method to get the indexed part of the extended header, since elsewhere we assume that header + extended header + data = file size, and we wouldn’t want to leave part of the file un-allocated.

AttributeError: module 'mrcfile' has no attribute 'open

I installed mrcfile by pip.

import mrcfile
went through well.

However,
with mrcfile.open('EMD-3197.map') as mrc:
resulted in

"AttributeError: module 'mrcfile' has no attribute 'open"

print (mrcfile.validate('EMD-3197.map'))
resulted in
AttributeError: module 'mrcfile' has no attribute 'validate'
as well.

/Users/kimd999/Library/Python/2.7/bin
that has
f2py, mrcfile-header, mrcfile-validate are in my PATH.

Can I know why I can't run mrcfile?
I can't run mrcfile both in my macOS (10.14.6) and Linux (centos).

Installation by
conda install --channel conda-forge mrcfile
also didn't help the situation.

Mach stamp for little endian?

Would you consider adding ('\x44', '\x11', '\x00', '\x00') as a little-endian mach stamp? I have the following exception for a decent number of maps in EMDB:

  File "/nfs/public/rw/pdbe/httpd-em/software/anaconda2/envs/meshmaker/lib/python2.7/site-packages/mrcfile/load_functions.py", line 153, in mmap
    return MrcMemmap(name, mode=mode, permissive=permissive)
  File "/nfs/public/rw/pdbe/httpd-em/software/anaconda2/envs/meshmaker/lib/python2.7/site-packages/mrcfile/mrcfile.py", line 105, in __init__
    self._read()
  File "/nfs/public/rw/pdbe/httpd-em/software/anaconda2/envs/meshmaker/lib/python2.7/site-packages/mrcfile/mrcfile.py", line 121, in _read
    super(MrcFile, self)._read()
  File "/nfs/public/rw/pdbe/httpd-em/software/anaconda2/envs/meshmaker/lib/python2.7/site-packages/mrcfile/mrcinterpreter.py", line 149, in _read
    self._read_header()
  File "/nfs/public/rw/pdbe/httpd-em/software/anaconda2/envs/meshmaker/lib/python2.7/site-packages/mrcfile/mrcinterpreter.py", line 186, in _read_header
    byte_order = utils.byte_order_from_machine_stamp(header.machst)
  File "/nfs/public/rw/pdbe/httpd-em/software/anaconda2/envs/meshmaker/lib/python2.7/site-packages/mrcfile/utils.py", line 190, in byte_order_from_machine_stamp
    raise ValueError("Unrecognised machine stamp: " + pretty_bytes)
ValueError: Unrecognised machine stamp: 0x44 0x11 0x00 0x00

FEI extended header implementation different from paper

The current implementation of the FEI extended header follows the specification in the EPU manual as linked in the MRC2014 home page.

However, the paper has a completely different version of the FEI extended header:

These files currently include a 128 bytes extended header entry per image, located between the main header and the data block, which includes additional metadata such as tilt angles, stage position, exposure time, etc. The extended header has a fixed size of 1024 entries if NZ ⩽ 1024 and 8192 entries if NZ > 1024.

This is the extended header as used in tomography, and unfortunately, opening files with such a header currently fails with mrcfile.

I'd suggest that the package implement the extended header as outlined in the paper, and that other FEI extended headers (like the one in EPU) get their own exttyp.

test_data_is_not_read_if_dimensions_are_too_huge fails on i586

Hi,

I get following failure:

[   21s] =================================== FAILURES ===================================
[   21s] ________ MrcMemmapTest.test_data_is_not_read_if_dimensions_are_too_huge ________
[   21s] 
[   21s] self = <tests.test_mrcmemmap.MrcMemmapTest testMethod=test_data_is_not_read_if_dimensions_are_too_huge>
[   21s] 
[   21s]     def test_data_is_not_read_if_dimensions_are_too_huge(self):
[   21s]         # Prepare x, y and z counts to try to trigger an out-of-memory error
[   21s]         # The individual values need to fit in int32 values to be stored in the header
[   21s]         # and their product must be as large as possible while still less than
[   21s]         # sys.maxsize (if larger, it triggers an index overflow error instead)
[   21s]         max_i4 = np.iinfo('i4').max
[   21s]         max_arr = sys.maxsize
[   21s]         nx = max_i4
[   21s]         ny = min(max_i4, max_arr // nx)
[   21s]         nz = min(max_i4, max_arr // (nx * ny))
[   21s]         # Check that an allocation of this size really does cause a memory error
[   21s]         with self.assertRaises(MemoryError):
[   21s]             _ = bytearray(nx * ny * nz)
[   21s]     
[   21s]         # Now put these values into a file
[   21s]         with self.newmrc(self.temp_mrc_name, mode='w+') as mrc:
[   21s]             mrc.set_data(np.arange(24, dtype=np.int8).reshape(2, 3, 4))
[   21s]             mrc.header.mx = mrc.header.nx = nx
[   21s]             mrc.header.my = mrc.header.ny = ny
[   21s]             mrc.header.mz = mrc.header.nz = nz
[   21s]     
[   21s]         # And now check that if we open the file, we avoid the problem and don't try
[   21s]         # to allocate enough memory to cause an out-of-memory error
[   21s]         with warnings.catch_warnings(record=True) as w:
[   21s]             warnings.simplefilter("always")
[   21s] >           with self.newmrc(self.temp_mrc_name, permissive=True) as mrc:
[   21s] 
[   21s] tests/test_mrcfile.py:352: 
[   21s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[   21s] mrcfile/mrcfile.py:115: in __init__
[   21s]     self._read(header_only)
[   21s] mrcfile/mrcfile.py:131: in _read
[   21s]     super(MrcFile, self)._read(header_only)
[   21s] mrcfile/mrcinterpreter.py:176: in _read
[   21s]     self._read_data()
[   21s] mrcfile/mrcmemmap.py:112: in _read_data
[   21s]     self._open_memmap(dtype, shape)
[   21s] mrcfile/mrcmemmap.py:121: in _open_memmap
[   21s]     self._data = np.memmap(self._iostream,
[   21s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[   21s] 
[   21s] subtype = <class 'numpy.memmap'>
[   21s] filename = <_io.BufferedReader name='/tmp/tmpbfu8zgrg/test_mrcfile.mrc'>
[   21s] dtype = dtype('int8'), mode = 'r', offset = 1024, shape = (1, 1, 2147483647)
[   21s] order = 'C'
[   21s] 
[   21s]     def __new__(subtype, filename, dtype=uint8, mode='r+', offset=0,
[   21s]                 shape=None, order='C'):
[   21s]         # Import here to minimize 'import numpy' overhead
[   21s]         import mmap
[   21s]         import os.path
[   21s]         try:
[   21s]             mode = mode_equivalents[mode]
[   21s]         except KeyError as e:
[   21s]             if mode not in valid_filemodes:
[   21s]                 raise ValueError(
[   21s]                     "mode must be one of {!r} (got {!r})"
[   21s]                     .format(valid_filemodes + list(mode_equivalents.keys()), mode)
[   21s]                 ) from None
[   21s]     
[   21s]         if mode == 'w+' and shape is None:
[   21s]             raise ValueError("shape must be given")
[   21s]     
[   21s]         if hasattr(filename, 'read'):
[   21s]             f_ctx = nullcontext(filename)
[   21s]         else:
[   21s]             f_ctx = open(os_fspath(filename), ('r' if mode == 'c' else mode)+'b')
[   21s]     
[   21s]         with f_ctx as fid:
[   21s]             fid.seek(0, 2)
[   21s]             flen = fid.tell()
[   21s]             descr = dtypedescr(dtype)
[   21s]             _dbytes = descr.itemsize
[   21s]     
[   21s]             if shape is None:
[   21s]                 bytes = flen - offset
[   21s]                 if bytes % _dbytes:
[   21s]                     raise ValueError("Size of available data is not a "
[   21s]                             "multiple of the data-type size.")
[   21s]                 size = bytes // _dbytes
[   21s]                 shape = (size,)
[   21s]             else:
[   21s]                 if not isinstance(shape, tuple):
[   21s]                     shape = (shape,)
[   21s]                 size = np.intp(1)  # avoid default choice of np.int_, which might overflow
[   21s]                 for k in shape:
[   21s]                     size *= k
[   21s]     
[   21s]             bytes = int(offset + size*_dbytes)
[   21s]     
[   21s]             if mode in ('w+', 'r+') and flen < bytes:
[   21s]                 fid.seek(bytes - 1, 0)
[   21s]                 fid.write(b'\0')
[   21s]                 fid.flush()
[   21s]     
[   21s]             if mode == 'c':
[   21s]                 acc = mmap.ACCESS_COPY
[   21s]             elif mode == 'r':
[   21s]                 acc = mmap.ACCESS_READ
[   21s]             else:
[   21s]                 acc = mmap.ACCESS_WRITE
[   21s]     
[   21s]             start = offset - offset % mmap.ALLOCATIONGRANULARITY
[   21s]             bytes -= start
[   21s]             array_offset = offset - start
[   21s] >           mm = mmap.mmap(fid.fileno(), bytes, access=acc, offset=start)
[   21s] E           OverflowError: memory mapped length must be positive
[   21s] 
[   21s] /usr/lib/python3.11/site-packages/numpy/core/memmap.py:267: OverflowError
[   21s] =============================== warnings summary ===============================

It would be very welcome if you would find time to look at it or would give me any pointers how to debug.

Cheers

Creating large mrcfile with memmap

I was trying to copy an existing HDF5 file to MRC format. The HDF5 file is too big. I could use the existing mechanism to dump the whole HDF5 dataset to an MRC file as:

with mrcfile.new('tmp.mrc') as mrc:
    mrc.set_data(large_hdf5[...], dtype=np.float32))

However, that requires me to load the whole HDF5 file into memory. A similar problem arises if I want to initialize an empty MRC file (similar to np.zeros) of size larger than RAM (e.g. 4000x4000x4000), it would require me to create np.zeros((4000,4000,4000)) beforehand, which is a bit redundant.

This can be solved using memmaps, which is what I tend to use when h5py is not available/needed.

By inspection/reverse engineering I managed to come up with the following code which will initialize an empty dataset without requiring to create it with numpy beforehand:

import mrcfile as mrc
import numpy as np

def newmrc(fpath, shape, dtype, fill=None):
    f = mrc.MrcMemmap(fpath, mode='w+', overwrite=True)
    dsize = np.prod(shape) * np.dtype(dtype).itemsize   # data size
    hsize = f.header.nbytes + f.header.nsymbt           # header size
    f._iostream.truncate(hsize + dsize)
    f._open_memmap(dtype, shape)
    f.update_header_from_data()
    f.update_header_stats()
    if fill is not None:
        f.data[...] = fill
    return f

That would allow you to create a memmaped MRC without having to hold a very large array in memory beforehand. You could then apply any chunking mechanism to write the full file, e.g.:

# array on disk full of zeros
f = newmrc('/tmp/test.mrc', (2000, 2000, 2000), np.float32, fill=0)

It will take time (a lot) to write a 30GB file, but it will do it without memory problems.

I thought it was useful to share the script, because 1) maybe I'm missing something important or 2) it might be useful to add to the library itself or 3) there might be a better alternative to what I want to achieve but I failed to find it.

8 test failures on s390x

mrcfile version 1.3.0 test suite is showing 8 test failures when executed on Fedora rawhide s390x arch:

$ python3 -m unittest tests
..............................FF.............................................................................................................................FF..........................................................................................................................................................................................................................................................................FF.................................................................................................................FF...................................................................................../builddir/build/BUILD/mrcfile-1.3.0/tests/test_utils.py:139: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  utils.mode_from_dtype(np.dtype(np.bool))
...................................................................................
======================================================================
FAIL: test_extended_header_from_FEI1_file (tests.test_bzip2mrcfile.Bzip2MrcFileTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/mrcfile-1.3.0/tests/test_mrcfile.py", line 177, in test_extended_header_from_FEI1_file
    assert ext[0]['Metadata size'] == 768
AssertionError
======================================================================
FAIL: test_extended_header_from_FEI2_file (tests.test_bzip2mrcfile.Bzip2MrcFileTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/mrcfile-1.3.0/tests/test_mrcfile.py", line 193, in test_extended_header_from_FEI2_file
    assert ext[0]['Metadata size'] == 888
AssertionError
======================================================================
FAIL: test_extended_header_from_FEI1_file (tests.test_gzipmrcfile.GzipMrcFileTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/mrcfile-1.3.0/tests/test_mrcfile.py", line 177, in test_extended_header_from_FEI1_file
    assert ext[0]['Metadata size'] == 768
AssertionError
======================================================================
FAIL: test_extended_header_from_FEI2_file (tests.test_gzipmrcfile.GzipMrcFileTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/mrcfile-1.3.0/tests/test_mrcfile.py", line 193, in test_extended_header_from_FEI2_file
    assert ext[0]['Metadata size'] == 888
AssertionError
======================================================================
FAIL: test_extended_header_from_FEI1_file (tests.test_mrcfile.MrcFileTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/mrcfile-1.3.0/tests/test_mrcfile.py", line 177, in test_extended_header_from_FEI1_file
    assert ext[0]['Metadata size'] == 768
AssertionError
======================================================================
FAIL: test_extended_header_from_FEI2_file (tests.test_mrcfile.MrcFileTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/mrcfile-1.3.0/tests/test_mrcfile.py", line 193, in test_extended_header_from_FEI2_file
    assert ext[0]['Metadata size'] == 888
AssertionError
======================================================================
FAIL: test_extended_header_from_FEI1_file (tests.test_mrcmemmap.MrcMemmapTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/mrcfile-1.3.0/tests/test_mrcfile.py", line 177, in test_extended_header_from_FEI1_file
    assert ext[0]['Metadata size'] == 768
AssertionError
======================================================================
FAIL: test_extended_header_from_FEI2_file (tests.test_mrcmemmap.MrcMemmapTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/mrcfile-1.3.0/tests/test_mrcfile.py", line 193, in test_extended_header_from_FEI2_file
    assert ext[0]['Metadata size'] == 888
AssertionError
----------------------------------------------------------------------
Ran 710 tests in 11.009s
FAILED (failures=8)

failed to read FEI2 ext header with mrcfile 1.4.3

I have an mrc movie file with 13 frames. NZ = 13, exttyp = fei2, nsymbt = 15360. In the mrcinterpreter code I can see the whole 15360 bytes are read into ext_header_arr.

When trying to read the header I get:

mrcinterpreter.py:295: RuntimeWarning: File has exttyp 'b'FEI2'' but the extended header cannot be interpreted as that type
warnings.warn("File has exttyp '{}' but the extended header "

Or more detailed:

self._extended_header.dtype = dtype
^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: When changing to a larger dtype, its size must be a divisor of the total size in bytes of the last axis of the array.

If I read the ext header myself I know that for a single block the metadata size = 888 bytes. I think the problem is that 15360 is not divisible by 888, hence there's probably an unused area after all 13 blocks.

bug: possibility of writing invalid headers yielding unreadable files

Hey @colinpalmer - as per our discussion:

In juglab/cryoCARE_pip#43 we saw that files were being written, copying the original header but not the extended header. The length of the extended header is stored in the header and as such the start of the data is incorrectly specified in the files mrcfile writes.

Proposal: check and update the header entry containing the number of bytes of the extended header when writing a file to avoid this problem in the future

Is there any way this can be used in a lazy way?

So I may not want to load all the data into ram, maybe just a subset, is there any way this can be accommodated?

Mismatch in supported modes between documentation and code

The documentation says "Currently accepted modes are 0, 1, 2, 4 and 6", but mode 12 (float16) is supported in the current version of the code.

mx,my and mz potentially being set wrong

For entries with non x,y,z axis ordering e.g. z,y,x running update_header_from_data sets mx equal to nx, my equal to ny etc.

However mx is declared as sampling along X axis of unit cell not in terms of columns like nx.
My interpretation of the spec is that this should set nx == mz when z is the col axis.
Am I misinterpreting the mrc specification or is this an issue with not taking axis ordering into account in mrcfile?

Example entry EMD-30289's primary map.

Regards
Ryan

`mrcfile.write()` cannot be used with `pathlib.Path` objects

>       if name.endswith('.gz'):
E       AttributeError: 'PosixPath' object has no attribute 'endswith'

Would you accept a PR which coerces name to str in the write function?

Tests fail with Python 3.11 and Numpy 1.24.1: AssertionError

Some tests start failing with Python 3.11 and Numpy 1.24.1.
See: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1028707

> FAIL: test_cannot_edit_extended_header_in_read_only_mode 
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/<<PKGBUILDDIR>>/tests/test_mrcfile.py", line 219, in test_cannot_edit_extended_header_in_read_only_mode
>     with self.assertRaisesRegex(ValueError, 'read-only'):
> AssertionError: ValueError not raised
> 
> ======================================================================
> FAIL: test_data_is_not_copied_unnecessarily (tests.test_bzip2mrcfile.Bzip2MrcFileTest)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/<<PKGBUILDDIR>>/tests/test_mrcobject.py", line 341, in test_data_is_not_copied_unnecessarily
>     assert self.mrcobject.data is data
> AssertionError
> 
> ======================================================================
> FAIL: test_cannot_edit_extended_header_in_read_only_mode (tests.test_gzipmrcfile.GzipMrcFileTest)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/<<PKGBUILDDIR>>/tests/test_mrcfile.py", line 219, in test_cannot_edit_extended_header_in_read_only_mode
>     with self.assertRaisesRegex(ValueError, 'read-only'):
> AssertionError: ValueError not raised
> 
> ======================================================================
> FAIL: test_data_is_not_copied_unnecessarily (tests.test_gzipmrcfile.GzipMrcFileTest)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/<<PKGBUILDDIR>>/tests/test_mrcobject.py", line 341, in test_data_is_not_copied_unnecessarily
>     assert self.mrcobject.data is data
> AssertionError
> 
> ======================================================================
> FAIL: test_data_is_not_copied_unnecessarily (tests.test_mrcobject.MrcObjectTest)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/<<PKGBUILDDIR>>/tests/test_mrcobject.py", line 341, in test_data_is_not_copied_unnecessarily
>     assert self.mrcobject.data is data
> AssertionError
> 
> ======================================================================
> FAIL: test_data_is_not_copied_unnecessarily (tests.test_mrcinterpreter.MrcInterpreterTest)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/<<PKGBUILDDIR>>/tests/test_mrcobject.py", line 341, in test_data_is_not_copied_unnecessarily
>     assert self.mrcobject.data is data
> AssertionError
> 
> ======================================================================
> FAIL: test_cannot_edit_extended_header_in_read_only_mode (tests.test_mrcfile.MrcFileTest)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/<<PKGBUILDDIR>>/tests/test_mrcfile.py", line 219, in test_cannot_edit_extended_header_in_read_only_mode
>     with self.assertRaisesRegex(ValueError, 'read-only'):
> AssertionError: ValueError not raised
> 
> ======================================================================
> FAIL: test_data_is_not_copied_unnecessarily (tests.test_mrcfile.MrcFileTest)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/<<PKGBUILDDIR>>/tests/test_mrcobject.py", line 341, in test_data_is_not_copied_unnecessarily
>     assert self.mrcobject.data is data
> AssertionError
> 
> ======================================================================
> FAIL: test_cannot_edit_extended_header_in_read_only_mode (tests.test_mrcmemmap.MrcMemmapTest)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/<<PKGBUILDDIR>>/tests/test_mrcfile.py", line 219, in test_cannot_edit_extended_header_in_read_only_mode
>     with self.assertRaisesRegex(ValueError, 'read-only'):
> AssertionError: ValueError not raised
>

After changing:

assert self.mrcobject.data is data

with:

assert np.array_equal(self.mrcobject.data, data)

in tests/test_mrcfile.py, all tests pass except "AssertionError: ValueError not raised" caused by a numpy bug (link)

I'm not sure about this change, though.

Kind Regards

How to use header information

Hi,

I used Chimear to superimpose two density maps (A, B, I fit A to B) and save the superimposed one (B) as a new mrc file (C). But when I use mrcfile package to read it (C), how could I use header information to ensure the mrc.data is superimposed? If I directly load mrc.data as an array, it is the same the mrc.data of the original map (B).

Thank you

Test failures with NumPy 2.0.0rc2

When running the test suite against NumPy 2.0.0rc2, I'm getting the following test failures:

$ python -m unittest
.........................E...................E........................E.............EEE..................E...........EEE..............................................E...................E........................E.............EEE..................E...........EEE.................................................E...........E...........................EEE...............E...............E...........E................................EEE..................E..........................E...................E...................E....E.............EEE..................E...........EEE.................................E....................E...................E....E.............EEE..................E...........EEE.......................................................................................F......................
======================================================================
ERROR: test_complex64_dtype_is_preserved_in_mode_4 (tests.test_bzip2mrcfile.Bzip2MrcFileTest.test_complex64_dtype_is_preserved_in_mode_4)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 316, in test_complex64_dtype_is_preserved_in_mode_4
    warnings.simplefilter("ignore", np.ComplexWarning)
                                    ^^^^^^^^^^^^^^^^^
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/__init__.py", line 410, in __getattr__
    raise AttributeError("module {!r} has no attribute "
AttributeError: module 'numpy' has no attribute 'ComplexWarning'

======================================================================
ERROR: test_header_byte_order_is_changed_by_data_with_opposite_order (tests.test_bzip2mrcfile.Bzip2MrcFileTest.test_header_byte_order_is_changed_by_data_with_opposite_order)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 374, in test_header_byte_order_is_changed_by_data_with_opposite_order
    self.mrcobject.set_data(data.newbyteorder())
                            ^^^^^^^^^^^^^^^^^
AttributeError: `newbyteorder` was removed from the ndarray class in NumPy 2.0. Use `arr.view(arr.dtype.newbyteorder(order))` instead.

======================================================================
ERROR: test_permissive_read_with_wrong_machine_stamp (tests.test_bzip2mrcfile.Bzip2MrcFileTest.test_permissive_read_with_wrong_machine_stamp)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcfile.py", line 710, in test_permissive_read_with_wrong_machine_stamp
    wrong_byte_order = mrc.header.mode.newbyteorder().dtype.byteorder
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: `newbyteorder` was removed from the ndarray class in NumPy 2.0. Use `arr.view(arr.dtype.newbyteorder(order))` instead.

======================================================================
ERROR: test_setting_origin_as_separate_fields (tests.test_bzip2mrcfile.Bzip2MrcFileTest.test_setting_origin_as_separate_fields)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 724, in test_setting_origin_as_separate_fields
    expected = np.core.records.fromrecords((1., -5., 128.),
               ^^^^^^^^^^^^^^^
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/core/__init__.py", line 31, in __getattr__
    _raise_warning(attr_name)
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/core/_utils.py", line 10, in _raise_warning
    warnings.warn(
DeprecationWarning: numpy.core is deprecated and has been renamed to numpy._core. The numpy._core namespace contains private NumPy internals and its use is discouraged, as NumPy internals can change without warning in any release. In practice, most real-world usage of numpy.core is to access functionality in the public NumPy API. If that is the case, use the public NumPy API. If not, you are using NumPy internals. If you would still like to access an internal attribute, use numpy._core.records.

======================================================================
ERROR: test_setting_origin_as_single_field (tests.test_bzip2mrcfile.Bzip2MrcFileTest.test_setting_origin_as_single_field)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 729, in test_setting_origin_as_single_field
    origin = np.core.records.fromrecords((-1., 1., 0.),
             ^^^^^^^^^^^^^^^
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/core/__init__.py", line 31, in __getattr__
    _raise_warning(attr_name)
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/core/_utils.py", line 10, in _raise_warning
    warnings.warn(
DeprecationWarning: numpy.core is deprecated and has been renamed to numpy._core. The numpy._core namespace contains private NumPy internals and its use is discouraged, as NumPy internals can change without warning in any release. In practice, most real-world usage of numpy.core is to access functionality in the public NumPy API. If that is the case, use the public NumPy API. If not, you are using NumPy internals. If you would still like to access an internal attribute, use numpy._core.records.

======================================================================
ERROR: test_setting_origin_as_single_tuple (tests.test_bzip2mrcfile.Bzip2MrcFileTest.test_setting_origin_as_single_tuple)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 737, in test_setting_origin_as_single_tuple
    expected = np.core.records.fromrecords(origin,
               ^^^^^^^^^^^^^^^
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/core/__init__.py", line 31, in __getattr__
    _raise_warning(attr_name)
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/core/_utils.py", line 10, in _raise_warning
    warnings.warn(
DeprecationWarning: numpy.core is deprecated and has been renamed to numpy._core. The numpy._core namespace contains private NumPy internals and its use is discouraged, as NumPy internals can change without warning in any release. In practice, most real-world usage of numpy.core is to access functionality in the public NumPy API. If that is the case, use the public NumPy API. If not, you are using NumPy internals. If you would still like to access an internal attribute, use numpy._core.records.

======================================================================
ERROR: test_warning_for_stats_with_inf (tests.test_bzip2mrcfile.Bzip2MrcFileTest.test_warning_for_stats_with_inf)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 515, in test_warning_for_stats_with_inf
    assert header.dmin == np.NINF
                          ^^^^^^^
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/__init__.py", line 397, in __getattr__
    raise AttributeError(
AttributeError: `np.NINF` was removed in the NumPy 2.0 release. Use `-np.inf` instead.

======================================================================
ERROR: test_writing_image_mode_4_big_endian (tests.test_bzip2mrcfile.Bzip2MrcFileTest.test_writing_image_mode_4_big_endian)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcfile.py", line 568, in test_writing_image_mode_4_big_endian
    warnings.simplefilter("ignore", np.ComplexWarning)
                                    ^^^^^^^^^^^^^^^^^
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/__init__.py", line 410, in __getattr__
    raise AttributeError("module {!r} has no attribute "
AttributeError: module 'numpy' has no attribute 'ComplexWarning'

======================================================================
ERROR: test_writing_image_mode_4_little_endian (tests.test_bzip2mrcfile.Bzip2MrcFileTest.test_writing_image_mode_4_little_endian)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcfile.py", line 560, in test_writing_image_mode_4_little_endian
    warnings.simplefilter("ignore", np.ComplexWarning)
                                    ^^^^^^^^^^^^^^^^^
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/__init__.py", line 410, in __getattr__
    raise AttributeError("module {!r} has no attribute "
AttributeError: module 'numpy' has no attribute 'ComplexWarning'

======================================================================
ERROR: test_writing_image_mode_4_native_byte_order (tests.test_bzip2mrcfile.Bzip2MrcFileTest.test_writing_image_mode_4_native_byte_order)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcfile.py", line 552, in test_writing_image_mode_4_native_byte_order
    warnings.simplefilter("ignore", np.ComplexWarning)
                                    ^^^^^^^^^^^^^^^^^
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/__init__.py", line 410, in __getattr__
    raise AttributeError("module {!r} has no attribute "
AttributeError: module 'numpy' has no attribute 'ComplexWarning'

======================================================================
ERROR: test_complex64_dtype_is_preserved_in_mode_4 (tests.test_gzipmrcfile.GzipMrcFileTest.test_complex64_dtype_is_preserved_in_mode_4)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 316, in test_complex64_dtype_is_preserved_in_mode_4
    warnings.simplefilter("ignore", np.ComplexWarning)
                                    ^^^^^^^^^^^^^^^^^
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/__init__.py", line 410, in __getattr__
    raise AttributeError("module {!r} has no attribute "
AttributeError: module 'numpy' has no attribute 'ComplexWarning'

======================================================================
ERROR: test_header_byte_order_is_changed_by_data_with_opposite_order (tests.test_gzipmrcfile.GzipMrcFileTest.test_header_byte_order_is_changed_by_data_with_opposite_order)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 374, in test_header_byte_order_is_changed_by_data_with_opposite_order
    self.mrcobject.set_data(data.newbyteorder())
                            ^^^^^^^^^^^^^^^^^
AttributeError: `newbyteorder` was removed from the ndarray class in NumPy 2.0. Use `arr.view(arr.dtype.newbyteorder(order))` instead.

======================================================================
ERROR: test_permissive_read_with_wrong_machine_stamp (tests.test_gzipmrcfile.GzipMrcFileTest.test_permissive_read_with_wrong_machine_stamp)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcfile.py", line 710, in test_permissive_read_with_wrong_machine_stamp
    wrong_byte_order = mrc.header.mode.newbyteorder().dtype.byteorder
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: `newbyteorder` was removed from the ndarray class in NumPy 2.0. Use `arr.view(arr.dtype.newbyteorder(order))` instead.

======================================================================
ERROR: test_setting_origin_as_separate_fields (tests.test_gzipmrcfile.GzipMrcFileTest.test_setting_origin_as_separate_fields)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 724, in test_setting_origin_as_separate_fields
    expected = np.core.records.fromrecords((1., -5., 128.),
               ^^^^^^^^^^^^^^^
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/core/__init__.py", line 31, in __getattr__
    _raise_warning(attr_name)
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/core/_utils.py", line 10, in _raise_warning
    warnings.warn(
DeprecationWarning: numpy.core is deprecated and has been renamed to numpy._core. The numpy._core namespace contains private NumPy internals and its use is discouraged, as NumPy internals can change without warning in any release. In practice, most real-world usage of numpy.core is to access functionality in the public NumPy API. If that is the case, use the public NumPy API. If not, you are using NumPy internals. If you would still like to access an internal attribute, use numpy._core.records.

======================================================================
ERROR: test_setting_origin_as_single_field (tests.test_gzipmrcfile.GzipMrcFileTest.test_setting_origin_as_single_field)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 729, in test_setting_origin_as_single_field
    origin = np.core.records.fromrecords((-1., 1., 0.),
             ^^^^^^^^^^^^^^^
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/core/__init__.py", line 31, in __getattr__
    _raise_warning(attr_name)
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/core/_utils.py", line 10, in _raise_warning
    warnings.warn(
DeprecationWarning: numpy.core is deprecated and has been renamed to numpy._core. The numpy._core namespace contains private NumPy internals and its use is discouraged, as NumPy internals can change without warning in any release. In practice, most real-world usage of numpy.core is to access functionality in the public NumPy API. If that is the case, use the public NumPy API. If not, you are using NumPy internals. If you would still like to access an internal attribute, use numpy._core.records.

======================================================================
ERROR: test_setting_origin_as_single_tuple (tests.test_gzipmrcfile.GzipMrcFileTest.test_setting_origin_as_single_tuple)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 737, in test_setting_origin_as_single_tuple
    expected = np.core.records.fromrecords(origin,
               ^^^^^^^^^^^^^^^
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/core/__init__.py", line 31, in __getattr__
    _raise_warning(attr_name)
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/core/_utils.py", line 10, in _raise_warning
    warnings.warn(
DeprecationWarning: numpy.core is deprecated and has been renamed to numpy._core. The numpy._core namespace contains private NumPy internals and its use is discouraged, as NumPy internals can change without warning in any release. In practice, most real-world usage of numpy.core is to access functionality in the public NumPy API. If that is the case, use the public NumPy API. If not, you are using NumPy internals. If you would still like to access an internal attribute, use numpy._core.records.

======================================================================
ERROR: test_warning_for_stats_with_inf (tests.test_gzipmrcfile.GzipMrcFileTest.test_warning_for_stats_with_inf)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 515, in test_warning_for_stats_with_inf
    assert header.dmin == np.NINF
                          ^^^^^^^
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/__init__.py", line 397, in __getattr__
    raise AttributeError(
AttributeError: `np.NINF` was removed in the NumPy 2.0 release. Use `-np.inf` instead.

======================================================================
ERROR: test_writing_image_mode_4_big_endian (tests.test_gzipmrcfile.GzipMrcFileTest.test_writing_image_mode_4_big_endian)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcfile.py", line 568, in test_writing_image_mode_4_big_endian
    warnings.simplefilter("ignore", np.ComplexWarning)
                                    ^^^^^^^^^^^^^^^^^
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/__init__.py", line 410, in __getattr__
    raise AttributeError("module {!r} has no attribute "
AttributeError: module 'numpy' has no attribute 'ComplexWarning'

======================================================================
ERROR: test_writing_image_mode_4_little_endian (tests.test_gzipmrcfile.GzipMrcFileTest.test_writing_image_mode_4_little_endian)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcfile.py", line 560, in test_writing_image_mode_4_little_endian
    warnings.simplefilter("ignore", np.ComplexWarning)
                                    ^^^^^^^^^^^^^^^^^
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/__init__.py", line 410, in __getattr__
    raise AttributeError("module {!r} has no attribute "
AttributeError: module 'numpy' has no attribute 'ComplexWarning'

======================================================================
ERROR: test_writing_image_mode_4_native_byte_order (tests.test_gzipmrcfile.GzipMrcFileTest.test_writing_image_mode_4_native_byte_order)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcfile.py", line 552, in test_writing_image_mode_4_native_byte_order
    warnings.simplefilter("ignore", np.ComplexWarning)
                                    ^^^^^^^^^^^^^^^^^
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/__init__.py", line 410, in __getattr__
    raise AttributeError("module {!r} has no attribute "
AttributeError: module 'numpy' has no attribute 'ComplexWarning'

======================================================================
ERROR: test_complex64_dtype_is_preserved_in_mode_4 (tests.test_mrcobject.MrcObjectTest.test_complex64_dtype_is_preserved_in_mode_4)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 316, in test_complex64_dtype_is_preserved_in_mode_4
    warnings.simplefilter("ignore", np.ComplexWarning)
                                    ^^^^^^^^^^^^^^^^^
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/__init__.py", line 410, in __getattr__
    raise AttributeError("module {!r} has no attribute "
AttributeError: module 'numpy' has no attribute 'ComplexWarning'

======================================================================
ERROR: test_header_byte_order_is_changed_by_data_with_opposite_order (tests.test_mrcobject.MrcObjectTest.test_header_byte_order_is_changed_by_data_with_opposite_order)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 374, in test_header_byte_order_is_changed_by_data_with_opposite_order
    self.mrcobject.set_data(data.newbyteorder())
                            ^^^^^^^^^^^^^^^^^
AttributeError: `newbyteorder` was removed from the ndarray class in NumPy 2.0. Use `arr.view(arr.dtype.newbyteorder(order))` instead.

======================================================================
ERROR: test_setting_origin_as_separate_fields (tests.test_mrcobject.MrcObjectTest.test_setting_origin_as_separate_fields)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 724, in test_setting_origin_as_separate_fields
    expected = np.core.records.fromrecords((1., -5., 128.),
               ^^^^^^^^^^^^^^^
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/core/__init__.py", line 31, in __getattr__
    _raise_warning(attr_name)
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/core/_utils.py", line 10, in _raise_warning
    warnings.warn(
DeprecationWarning: numpy.core is deprecated and has been renamed to numpy._core. The numpy._core namespace contains private NumPy internals and its use is discouraged, as NumPy internals can change without warning in any release. In practice, most real-world usage of numpy.core is to access functionality in the public NumPy API. If that is the case, use the public NumPy API. If not, you are using NumPy internals. If you would still like to access an internal attribute, use numpy._core.records.

======================================================================
ERROR: test_setting_origin_as_single_field (tests.test_mrcobject.MrcObjectTest.test_setting_origin_as_single_field)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 729, in test_setting_origin_as_single_field
    origin = np.core.records.fromrecords((-1., 1., 0.),
             ^^^^^^^^^^^^^^^
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/core/__init__.py", line 31, in __getattr__
    _raise_warning(attr_name)
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/core/_utils.py", line 10, in _raise_warning
    warnings.warn(
DeprecationWarning: numpy.core is deprecated and has been renamed to numpy._core. The numpy._core namespace contains private NumPy internals and its use is discouraged, as NumPy internals can change without warning in any release. In practice, most real-world usage of numpy.core is to access functionality in the public NumPy API. If that is the case, use the public NumPy API. If not, you are using NumPy internals. If you would still like to access an internal attribute, use numpy._core.records.

======================================================================
ERROR: test_setting_origin_as_single_tuple (tests.test_mrcobject.MrcObjectTest.test_setting_origin_as_single_tuple)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 737, in test_setting_origin_as_single_tuple
    expected = np.core.records.fromrecords(origin,
               ^^^^^^^^^^^^^^^
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/core/__init__.py", line 31, in __getattr__
    _raise_warning(attr_name)
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/core/_utils.py", line 10, in _raise_warning
    warnings.warn(
DeprecationWarning: numpy.core is deprecated and has been renamed to numpy._core. The numpy._core namespace contains private NumPy internals and its use is discouraged, as NumPy internals can change without warning in any release. In practice, most real-world usage of numpy.core is to access functionality in the public NumPy API. If that is the case, use the public NumPy API. If not, you are using NumPy internals. If you would still like to access an internal attribute, use numpy._core.records.

======================================================================
ERROR: test_warning_for_stats_with_inf (tests.test_mrcobject.MrcObjectTest.test_warning_for_stats_with_inf)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 515, in test_warning_for_stats_with_inf
    assert header.dmin == np.NINF
                          ^^^^^^^
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/__init__.py", line 397, in __getattr__
    raise AttributeError(
AttributeError: `np.NINF` was removed in the NumPy 2.0 release. Use `-np.inf` instead.

======================================================================
ERROR: test_complex64_dtype_is_preserved_in_mode_4 (tests.test_mrcinterpreter.MrcInterpreterTest.test_complex64_dtype_is_preserved_in_mode_4)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 316, in test_complex64_dtype_is_preserved_in_mode_4
    warnings.simplefilter("ignore", np.ComplexWarning)
                                    ^^^^^^^^^^^^^^^^^
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/__init__.py", line 410, in __getattr__
    raise AttributeError("module {!r} has no attribute "
AttributeError: module 'numpy' has no attribute 'ComplexWarning'

======================================================================
ERROR: test_header_byte_order_is_changed_by_data_with_opposite_order (tests.test_mrcinterpreter.MrcInterpreterTest.test_header_byte_order_is_changed_by_data_with_opposite_order)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 374, in test_header_byte_order_is_changed_by_data_with_opposite_order
    self.mrcobject.set_data(data.newbyteorder())
                            ^^^^^^^^^^^^^^^^^
AttributeError: `newbyteorder` was removed from the ndarray class in NumPy 2.0. Use `arr.view(arr.dtype.newbyteorder(order))` instead.

======================================================================
ERROR: test_setting_origin_as_separate_fields (tests.test_mrcinterpreter.MrcInterpreterTest.test_setting_origin_as_separate_fields)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 724, in test_setting_origin_as_separate_fields
    expected = np.core.records.fromrecords((1., -5., 128.),
               ^^^^^^^^^^^^^^^
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/core/__init__.py", line 31, in __getattr__
    _raise_warning(attr_name)
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/core/_utils.py", line 10, in _raise_warning
    warnings.warn(
DeprecationWarning: numpy.core is deprecated and has been renamed to numpy._core. The numpy._core namespace contains private NumPy internals and its use is discouraged, as NumPy internals can change without warning in any release. In practice, most real-world usage of numpy.core is to access functionality in the public NumPy API. If that is the case, use the public NumPy API. If not, you are using NumPy internals. If you would still like to access an internal attribute, use numpy._core.records.

======================================================================
ERROR: test_setting_origin_as_single_field (tests.test_mrcinterpreter.MrcInterpreterTest.test_setting_origin_as_single_field)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 729, in test_setting_origin_as_single_field
    origin = np.core.records.fromrecords((-1., 1., 0.),
             ^^^^^^^^^^^^^^^
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/core/__init__.py", line 31, in __getattr__
    _raise_warning(attr_name)
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/core/_utils.py", line 10, in _raise_warning
    warnings.warn(
DeprecationWarning: numpy.core is deprecated and has been renamed to numpy._core. The numpy._core namespace contains private NumPy internals and its use is discouraged, as NumPy internals can change without warning in any release. In practice, most real-world usage of numpy.core is to access functionality in the public NumPy API. If that is the case, use the public NumPy API. If not, you are using NumPy internals. If you would still like to access an internal attribute, use numpy._core.records.

======================================================================
ERROR: test_setting_origin_as_single_tuple (tests.test_mrcinterpreter.MrcInterpreterTest.test_setting_origin_as_single_tuple)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 737, in test_setting_origin_as_single_tuple
    expected = np.core.records.fromrecords(origin,
               ^^^^^^^^^^^^^^^
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/core/__init__.py", line 31, in __getattr__
    _raise_warning(attr_name)
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/core/_utils.py", line 10, in _raise_warning
    warnings.warn(
DeprecationWarning: numpy.core is deprecated and has been renamed to numpy._core. The numpy._core namespace contains private NumPy internals and its use is discouraged, as NumPy internals can change without warning in any release. In practice, most real-world usage of numpy.core is to access functionality in the public NumPy API. If that is the case, use the public NumPy API. If not, you are using NumPy internals. If you would still like to access an internal attribute, use numpy._core.records.

======================================================================
ERROR: test_warning_for_stats_with_inf (tests.test_mrcinterpreter.MrcInterpreterTest.test_warning_for_stats_with_inf)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 515, in test_warning_for_stats_with_inf
    assert header.dmin == np.NINF
                          ^^^^^^^
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/__init__.py", line 397, in __getattr__
    raise AttributeError(
AttributeError: `np.NINF` was removed in the NumPy 2.0 release. Use `-np.inf` instead.

======================================================================
ERROR: test_complex64_dtype_is_preserved_in_mode_4 (tests.test_mrcfile.MrcFileTest.test_complex64_dtype_is_preserved_in_mode_4)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 316, in test_complex64_dtype_is_preserved_in_mode_4
    warnings.simplefilter("ignore", np.ComplexWarning)
                                    ^^^^^^^^^^^^^^^^^
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/__init__.py", line 410, in __getattr__
    raise AttributeError("module {!r} has no attribute "
AttributeError: module 'numpy' has no attribute 'ComplexWarning'

======================================================================
ERROR: test_header_byte_order_is_changed_by_data_with_opposite_order (tests.test_mrcfile.MrcFileTest.test_header_byte_order_is_changed_by_data_with_opposite_order)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 374, in test_header_byte_order_is_changed_by_data_with_opposite_order
    self.mrcobject.set_data(data.newbyteorder())
                            ^^^^^^^^^^^^^^^^^
AttributeError: `newbyteorder` was removed from the ndarray class in NumPy 2.0. Use `arr.view(arr.dtype.newbyteorder(order))` instead.

======================================================================
ERROR: test_non_mrc_file_gives_correct_warnings_in_permissive_mode (tests.test_mrcfile.MrcFileTest.test_non_mrc_file_gives_correct_warnings_in_permissive_mode)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/mrcfile/mrcinterpreter.py", line 228, in _read_header
    utils.dtype_from_mode(header.mode)
  File "/tmp/mrcfile/mrcfile/utils.py", line 176, in dtype_from_mode
    raise ValueError("Unrecognised mode '{0}'".format(mode))
ValueError: Unrecognised mode '1380206665'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcfile.py", line 116, in test_non_mrc_file_gives_correct_warnings_in_permissive_mode
    with self.newmrc(name, permissive=True) as mrc:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/mrcfile/mrcfile/mrcfile.py", line 115, in __init__
    self._read(header_only)
  File "/tmp/mrcfile/mrcfile/mrcfile.py", line 131, in _read
    super(MrcFile, self)._read(header_only)
  File "/tmp/mrcfile/mrcfile/mrcinterpreter.py", line 170, in _read
    self._read_header()
  File "/tmp/mrcfile/mrcfile/mrcinterpreter.py", line 231, in _read_header
    utils.dtype_from_mode(header.mode.newbyteorder())
                          ^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: `newbyteorder` was removed from the ndarray class in NumPy 2.0. Use `arr.view(arr.dtype.newbyteorder(order))` instead.

======================================================================
ERROR: test_permissive_read_with_wrong_machine_stamp (tests.test_mrcfile.MrcFileTest.test_permissive_read_with_wrong_machine_stamp)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcfile.py", line 710, in test_permissive_read_with_wrong_machine_stamp
    wrong_byte_order = mrc.header.mode.newbyteorder().dtype.byteorder
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: `newbyteorder` was removed from the ndarray class in NumPy 2.0. Use `arr.view(arr.dtype.newbyteorder(order))` instead.

======================================================================
ERROR: test_setting_origin_as_separate_fields (tests.test_mrcfile.MrcFileTest.test_setting_origin_as_separate_fields)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 724, in test_setting_origin_as_separate_fields
    expected = np.core.records.fromrecords((1., -5., 128.),
               ^^^^^^^^^^^^^^^
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/core/__init__.py", line 31, in __getattr__
    _raise_warning(attr_name)
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/core/_utils.py", line 10, in _raise_warning
    warnings.warn(
DeprecationWarning: numpy.core is deprecated and has been renamed to numpy._core. The numpy._core namespace contains private NumPy internals and its use is discouraged, as NumPy internals can change without warning in any release. In practice, most real-world usage of numpy.core is to access functionality in the public NumPy API. If that is the case, use the public NumPy API. If not, you are using NumPy internals. If you would still like to access an internal attribute, use numpy._core.records.

======================================================================
ERROR: test_setting_origin_as_single_field (tests.test_mrcfile.MrcFileTest.test_setting_origin_as_single_field)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 729, in test_setting_origin_as_single_field
    origin = np.core.records.fromrecords((-1., 1., 0.),
             ^^^^^^^^^^^^^^^
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/core/__init__.py", line 31, in __getattr__
    _raise_warning(attr_name)
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/core/_utils.py", line 10, in _raise_warning
    warnings.warn(
DeprecationWarning: numpy.core is deprecated and has been renamed to numpy._core. The numpy._core namespace contains private NumPy internals and its use is discouraged, as NumPy internals can change without warning in any release. In practice, most real-world usage of numpy.core is to access functionality in the public NumPy API. If that is the case, use the public NumPy API. If not, you are using NumPy internals. If you would still like to access an internal attribute, use numpy._core.records.

======================================================================
ERROR: test_setting_origin_as_single_tuple (tests.test_mrcfile.MrcFileTest.test_setting_origin_as_single_tuple)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 737, in test_setting_origin_as_single_tuple
    expected = np.core.records.fromrecords(origin,
               ^^^^^^^^^^^^^^^
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/core/__init__.py", line 31, in __getattr__
    _raise_warning(attr_name)
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/core/_utils.py", line 10, in _raise_warning
    warnings.warn(
DeprecationWarning: numpy.core is deprecated and has been renamed to numpy._core. The numpy._core namespace contains private NumPy internals and its use is discouraged, as NumPy internals can change without warning in any release. In practice, most real-world usage of numpy.core is to access functionality in the public NumPy API. If that is the case, use the public NumPy API. If not, you are using NumPy internals. If you would still like to access an internal attribute, use numpy._core.records.

======================================================================
ERROR: test_warning_for_stats_with_inf (tests.test_mrcfile.MrcFileTest.test_warning_for_stats_with_inf)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 515, in test_warning_for_stats_with_inf
    assert header.dmin == np.NINF
                          ^^^^^^^
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/__init__.py", line 397, in __getattr__
    raise AttributeError(
AttributeError: `np.NINF` was removed in the NumPy 2.0 release. Use `-np.inf` instead.

======================================================================
ERROR: test_writing_image_mode_4_big_endian (tests.test_mrcfile.MrcFileTest.test_writing_image_mode_4_big_endian)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcfile.py", line 568, in test_writing_image_mode_4_big_endian
    warnings.simplefilter("ignore", np.ComplexWarning)
                                    ^^^^^^^^^^^^^^^^^
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/__init__.py", line 410, in __getattr__
    raise AttributeError("module {!r} has no attribute "
AttributeError: module 'numpy' has no attribute 'ComplexWarning'

======================================================================
ERROR: test_writing_image_mode_4_little_endian (tests.test_mrcfile.MrcFileTest.test_writing_image_mode_4_little_endian)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcfile.py", line 560, in test_writing_image_mode_4_little_endian
    warnings.simplefilter("ignore", np.ComplexWarning)
                                    ^^^^^^^^^^^^^^^^^
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/__init__.py", line 410, in __getattr__
    raise AttributeError("module {!r} has no attribute "
AttributeError: module 'numpy' has no attribute 'ComplexWarning'

======================================================================
ERROR: test_writing_image_mode_4_native_byte_order (tests.test_mrcfile.MrcFileTest.test_writing_image_mode_4_native_byte_order)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcfile.py", line 552, in test_writing_image_mode_4_native_byte_order
    warnings.simplefilter("ignore", np.ComplexWarning)
                                    ^^^^^^^^^^^^^^^^^
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/__init__.py", line 410, in __getattr__
    raise AttributeError("module {!r} has no attribute "
AttributeError: module 'numpy' has no attribute 'ComplexWarning'

======================================================================
ERROR: test_complex64_dtype_is_preserved_in_mode_4 (tests.test_mrcmemmap.MrcMemmapTest.test_complex64_dtype_is_preserved_in_mode_4)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 316, in test_complex64_dtype_is_preserved_in_mode_4
    warnings.simplefilter("ignore", np.ComplexWarning)
                                    ^^^^^^^^^^^^^^^^^
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/__init__.py", line 410, in __getattr__
    raise AttributeError("module {!r} has no attribute "
AttributeError: module 'numpy' has no attribute 'ComplexWarning'

======================================================================
ERROR: test_header_byte_order_is_changed_by_data_with_opposite_order (tests.test_mrcmemmap.MrcMemmapTest.test_header_byte_order_is_changed_by_data_with_opposite_order)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 374, in test_header_byte_order_is_changed_by_data_with_opposite_order
    self.mrcobject.set_data(data.newbyteorder())
                            ^^^^^^^^^^^^^^^^^
AttributeError: `newbyteorder` was removed from the ndarray class in NumPy 2.0. Use `arr.view(arr.dtype.newbyteorder(order))` instead.

======================================================================
ERROR: test_non_mrc_file_gives_correct_warnings_in_permissive_mode (tests.test_mrcmemmap.MrcMemmapTest.test_non_mrc_file_gives_correct_warnings_in_permissive_mode)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/mrcfile/mrcinterpreter.py", line 228, in _read_header
    utils.dtype_from_mode(header.mode)
  File "/tmp/mrcfile/mrcfile/utils.py", line 176, in dtype_from_mode
    raise ValueError("Unrecognised mode '{0}'".format(mode))
ValueError: Unrecognised mode '1380206665'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcfile.py", line 116, in test_non_mrc_file_gives_correct_warnings_in_permissive_mode
    with self.newmrc(name, permissive=True) as mrc:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/mrcfile/mrcfile/mrcfile.py", line 115, in __init__
    self._read(header_only)
  File "/tmp/mrcfile/mrcfile/mrcfile.py", line 131, in _read
    super(MrcFile, self)._read(header_only)
  File "/tmp/mrcfile/mrcfile/mrcinterpreter.py", line 170, in _read
    self._read_header()
  File "/tmp/mrcfile/mrcfile/mrcinterpreter.py", line 231, in _read_header
    utils.dtype_from_mode(header.mode.newbyteorder())
                          ^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: `newbyteorder` was removed from the ndarray class in NumPy 2.0. Use `arr.view(arr.dtype.newbyteorder(order))` instead.

======================================================================
ERROR: test_permissive_read_with_wrong_machine_stamp (tests.test_mrcmemmap.MrcMemmapTest.test_permissive_read_with_wrong_machine_stamp)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcfile.py", line 710, in test_permissive_read_with_wrong_machine_stamp
    wrong_byte_order = mrc.header.mode.newbyteorder().dtype.byteorder
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: `newbyteorder` was removed from the ndarray class in NumPy 2.0. Use `arr.view(arr.dtype.newbyteorder(order))` instead.

======================================================================
ERROR: test_setting_origin_as_separate_fields (tests.test_mrcmemmap.MrcMemmapTest.test_setting_origin_as_separate_fields)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 724, in test_setting_origin_as_separate_fields
    expected = np.core.records.fromrecords((1., -5., 128.),
               ^^^^^^^^^^^^^^^
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/core/__init__.py", line 31, in __getattr__
    _raise_warning(attr_name)
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/core/_utils.py", line 10, in _raise_warning
    warnings.warn(
DeprecationWarning: numpy.core is deprecated and has been renamed to numpy._core. The numpy._core namespace contains private NumPy internals and its use is discouraged, as NumPy internals can change without warning in any release. In practice, most real-world usage of numpy.core is to access functionality in the public NumPy API. If that is the case, use the public NumPy API. If not, you are using NumPy internals. If you would still like to access an internal attribute, use numpy._core.records.

======================================================================
ERROR: test_setting_origin_as_single_field (tests.test_mrcmemmap.MrcMemmapTest.test_setting_origin_as_single_field)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 729, in test_setting_origin_as_single_field
    origin = np.core.records.fromrecords((-1., 1., 0.),
             ^^^^^^^^^^^^^^^
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/core/__init__.py", line 31, in __getattr__
    _raise_warning(attr_name)
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/core/_utils.py", line 10, in _raise_warning
    warnings.warn(
DeprecationWarning: numpy.core is deprecated and has been renamed to numpy._core. The numpy._core namespace contains private NumPy internals and its use is discouraged, as NumPy internals can change without warning in any release. In practice, most real-world usage of numpy.core is to access functionality in the public NumPy API. If that is the case, use the public NumPy API. If not, you are using NumPy internals. If you would still like to access an internal attribute, use numpy._core.records.

======================================================================
ERROR: test_setting_origin_as_single_tuple (tests.test_mrcmemmap.MrcMemmapTest.test_setting_origin_as_single_tuple)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 737, in test_setting_origin_as_single_tuple
    expected = np.core.records.fromrecords(origin,
               ^^^^^^^^^^^^^^^
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/core/__init__.py", line 31, in __getattr__
    _raise_warning(attr_name)
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/core/_utils.py", line 10, in _raise_warning
    warnings.warn(
DeprecationWarning: numpy.core is deprecated and has been renamed to numpy._core. The numpy._core namespace contains private NumPy internals and its use is discouraged, as NumPy internals can change without warning in any release. In practice, most real-world usage of numpy.core is to access functionality in the public NumPy API. If that is the case, use the public NumPy API. If not, you are using NumPy internals. If you would still like to access an internal attribute, use numpy._core.records.

======================================================================
ERROR: test_warning_for_stats_with_inf (tests.test_mrcmemmap.MrcMemmapTest.test_warning_for_stats_with_inf)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 515, in test_warning_for_stats_with_inf
    assert header.dmin == np.NINF
                          ^^^^^^^
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/__init__.py", line 397, in __getattr__
    raise AttributeError(
AttributeError: `np.NINF` was removed in the NumPy 2.0 release. Use `-np.inf` instead.

======================================================================
ERROR: test_writing_image_mode_4_big_endian (tests.test_mrcmemmap.MrcMemmapTest.test_writing_image_mode_4_big_endian)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcfile.py", line 568, in test_writing_image_mode_4_big_endian
    warnings.simplefilter("ignore", np.ComplexWarning)
                                    ^^^^^^^^^^^^^^^^^
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/__init__.py", line 410, in __getattr__
    raise AttributeError("module {!r} has no attribute "
AttributeError: module 'numpy' has no attribute 'ComplexWarning'

======================================================================
ERROR: test_writing_image_mode_4_little_endian (tests.test_mrcmemmap.MrcMemmapTest.test_writing_image_mode_4_little_endian)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcfile.py", line 560, in test_writing_image_mode_4_little_endian
    warnings.simplefilter("ignore", np.ComplexWarning)
                                    ^^^^^^^^^^^^^^^^^
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/__init__.py", line 410, in __getattr__
    raise AttributeError("module {!r} has no attribute "
AttributeError: module 'numpy' has no attribute 'ComplexWarning'

======================================================================
ERROR: test_writing_image_mode_4_native_byte_order (tests.test_mrcmemmap.MrcMemmapTest.test_writing_image_mode_4_native_byte_order)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcfile.py", line 552, in test_writing_image_mode_4_native_byte_order
    warnings.simplefilter("ignore", np.ComplexWarning)
                                    ^^^^^^^^^^^^^^^^^
  File "/tmp/mrcfile/.tox/py312-numpy1.26/lib/python3.12/site-packages/numpy/__init__.py", line 410, in __getattr__
    raise AttributeError("module {!r} has no attribute "
AttributeError: module 'numpy' has no attribute 'ComplexWarning'

======================================================================
FAIL: test_invalid_mode (tests.test_validation.ValidationTest.test_invalid_mode)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_validation.py", line 182, in test_invalid_mode
    self.check_temp_mrc_invalid_with_warning("mode")
  File "/tmp/mrcfile/tests/test_validation.py", line 142, in check_temp_mrc_invalid_with_warning
    assert len(w) == 1
           ^^^^^^^^^^^
AssertionError

----------------------------------------------------------------------
Ran 811 tests in 8.425s

FAILED (failures=1, errors=54)

Modify `n*start`

Hi,
Thanks for a very useful package.
I was wondering whether you could add a way to modify n*start params (nxstart, nystart, nzstart) in a manner similar to voxel_size.
Thanks.
Paul

Writing/reading extended header seems to be broken in latest version

Hi @colinpalmer

I am having an issue with the latest version of mrcfile on pypi. It looks like there is some issue with the extended header being written.

If I write an mrcfile and then try to read the file back, the extended header does not seem to be interpreted correctly. Instead of an array of records I just get something like "[b'\x00' b'\x00' b'\x00' b'\x00' b'\x00' b'\x00' b'\x00' b'\x00' b'\x00' ... ] as the extended header.

Indeed, it looks like the bug existed before this version. However, previously it only affected files not written using the new_mmap function. I never noticed this because I have been writing mrcfile using new_mmap; however, in the new version of mrcfile the behaviour is consistent but unfortunately consistently broken!

Here is some python code to reproduce the problem:

import mrcfile
import numpy as np

def write():

    N = 10
    header_dtype = mrcfile.dtypes.get_ext_header_dtype(b"FEI1")
    
    data = np.zeros((N, 10, 10), dtype="float16")
    header = np.zeros(N, dtype=header_dtype)

    handle = mrcfile.new_mmap(
        "test_mmap.mrc",
        shape=data.shape,
        mrc_mode=mrcfile.utils.mode_from_dtype(data.dtype),
        overwrite=True,
        extended_header=header,
        exttyp="FEI1",
    )
    
    handle = mrcfile.new("test.mrc", overwrite=True)
    handle.set_data(data)
    handle.set_extended_header(header)



def read():
    
    print("TEST: test_mmap.mrc")
    handle = mrcfile.open("test_mmap.mrc")
    print(handle.extended_header.dtype)
    print(handle.extended_header)
    print("") 
    print("TEST: test.mrc")
    handle = mrcfile.open("test.mrc")
    print(handle.extended_header.dtype)
    print(handle.extended_header)


if __name__ == '__main__':
    write()
    read()

In version v1.4.3 when reading test_mmap.mrc I get the following output:

TEST: test_mmap.mrc
[('Metadata size', '<i4'), ('Metadata version', '<i4'), ('Bitmask 1', '<u4'), ('Timestamp', '<f8'), ('Microscope type', 'S16'), ('D-Number', 'S16'), ('Application', 'S16'), ('Application version', 'S16'), ('HT', '<f8'), ('Dose', '<f8'), ('Alpha tilt', '<f8'), ('Beta tilt', '<f8'), ('X-Stage', '<f8'), ('Y-Stage', '<f8'), ('Z-Stage', '<f8'), ('Tilt axis angle', '<f8'), ('Dual axis rotation', '<f8'), ('Pixel size X', '<f8'), ('Pixel size Y', '<f8'), ('Unused range', 'S48'), ('Defocus', '<f8'), ('STEM Defocus', '<f8'), ('Applied defocus', '<f8'), ('Instrument mode', '<i4'), ('Projection mode', '<i4'), ('Objective lens mode', 'S16'), ('High magnification mode', 'S16'), ('Probe mode', '<i4'), ('EFTEM On', '?'), ('Magnification', '<f8'), ('Bitmask 2', '<u4'), ('Camera length', '<f8'), ('Spot index', '<i4'), ('Illuminated area', '<f8'), ('Intensity', '<f8'), ('Convergence angle', '<f8'), ('Illumination mode', 'S16'), ('Wide convergence angle range', '?'), ('Slit inserted', '?'), ('Slit width', '<f8'), ('Acceleration voltage offset', '<f8'), ('Drift tube voltage', '<f8'), ('Energy shift', '<f8'), ('Shift offset X', '<f8'), ('Shift offset Y', '<f8'), ('Shift X', '<f8'), ('Shift Y', '<f8'), ('Integration time', '<f8'), ('Binning Width', '<i4'), ('Binning Height', '<i4'), ('Camera name', 'S16'), ('Readout area left', '<i4'), ('Readout area top', '<i4'), ('Readout area right', '<i4'), ('Readout area bottom', '<i4'), ('Ceta noise reduction', '?'), ('Ceta frames summed', '<i4'), ('Direct detector electron counting', '?'), ('Direct detector align frames', '?'), ('Camera param reserved 0', '<i4'), ('Camera param reserved 1', '<i4'), ('Camera param reserved 2', '<i4'), ('Camera param reserved 3', '<i4'), ('Bitmask 3', '<u4'), ('Camera param reserved 4', '<i4'), ('Camera param reserved 5', '<i4'), ('Camera param reserved 6', '<i4'), ('Camera param reserved 7', '<i4'), ('Camera param reserved 8', '<i4'), ('Camera param reserved 9', '<i4'), ('Phase Plate', '?'), ('STEM Detector name', 'S16'), ('Gain', '<f8'), ('Offset', '<f8'), ('STEM param reserved 0', '<i4'), ('STEM param reserved 1', '<i4'), ('STEM param reserved 2', '<i4'), ('STEM param reserved 3', '<i4'), ('STEM param reserved 4', '<i4'), ('Dwell time', '<f8'), ('Frame time', '<f8'), ('Scan size left', '<i4'), ('Scan size top', '<i4'), ('Scan size right', '<i4'), ('Scan size bottom', '<i4'), ('Full scan FOV X', '<f8'), ('Full scan FOV Y', '<f8'), ('Element', 'S16'), ('Energy interval lower', '<f8'), ('Energy interval higher', '<f8'), ('Method', '<i4'), ('Is dose fraction', '?'), ('Fraction number', '<i4'), ('Start frame', '<i4'), ('End frame', '<i4'), ('Input stack filename', 'S80'), ('Bitmask 4', '<u4'), ('Alpha tilt min', '<f8'), ('Alpha tilt max', '<f8')]
[(0, 0, 0, 0., b'', b'', b'', b'', 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., b'', 0., 0., 0., 0, 0, b'', b'', 0, False, 0., 0, 0., 0, 0., 0., 0., b'', False, False, 0., 0., 0., 0., 0., 0., 0., 0., 0., 0, 0, b'', 0, 0, 0, 0, False, 0, False, False, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, False, b'', 0., 0., 0, 0, 0, 0, 0, 0., 0., 0, 0, 0, 0, 0., 0., b'', 0., 0., 0, False, 0, 0, 0, b'', 0, 0., 0.)]

TEST: test.mrc
|V1
[b'\x00' b'\x00' b'\x00' b'\x00' b'\x00' b'\x00' b'\x00' b'\x00' b'\x00' ... ] # truncated for brevity

In version 1.5.0 I get this:

TEST: test_mmap.mrc
|V1
[b'\x00' b'\x00' b'\x00' b'\x00' b'\x00' b'\x00' b'\x00' b'\x00' b'\x00' ... ] # truncated for brevity

TEST: test.mrc
|V1
[b'\x00' b'\x00' b'\x00' b'\x00' b'\x00' b'\x00' b'\x00' b'\x00' b'\x00' ... ] # truncated for brevity

Gemmi vs mrcfile - values of some voxels are not equal

Dear all,

First of all, thank you for developing this package, it really comes in handy while reading big maps.

Recently, I've noticed that values of some voxels provided by gemmi and mrcfile are not equal while reading the same map.
Could you think of why it can happen?

Map file: EMD-1832.zip

from pathlib import Path
from typing import Tuple
import gemmi
import mrcfile
import numpy as np


def get_np_arr_from_gemmi(map_path: Path) -> np.ndarray:
    path = str(map_path.resolve())
    m = gemmi.read_ccp4_map(path)
    arr = np.array(m.grid)
    print(f'gemmi arr: dtype={arr.dtype}')
    return arr

def get_np_arr_from_mrcfile(map_path: Path) -> np.ndarray:
    path = str(map_path.resolve())
    m = mrcfile.open(path, mode='r')
    arr = m.data
    print(f'mrcfile arr: dtype={arr.dtype}')
    return arr

def compare_two_arrays(arr1, arr2):
    mask = arr1 == arr2
    reversed_mask = np.invert(mask)
    print(f'Not equal indices: {reversed_mask.nonzero()}')

def get_value_from_two_arrs(arr1, arr2, index: Tuple[int,int,int]):
    val_1 = arr1[index[0], index[1], index[2]]
    val_2 = arr2[index[0], index[1], index[2]]
    return val_1, val_2

if __name__ == '__main__':
    MAP_FILEPATH = Path('EMD-1832.map')
    arr1 = get_np_arr_from_gemmi(MAP_FILEPATH)
    arr2 = get_np_arr_from_mrcfile(MAP_FILEPATH)
    compare_two_arrays(arr1, arr2)
    # try to get value of (62, 31, 31) voxel
    indices = (62, 31, 31)
    gemmi_val, mrcfile_val = get_value_from_two_arrs(arr1, arr2, indices)
    print(f'indices: {indices}')
    print(f'gemmi_val {gemmi_val}')
    print(f'mrcfile_val {mrcfile_val}')

    indices = (1, 31, 32)
    gemmi_val, mrcfile_val = get_value_from_two_arrs(arr1, arr2, indices)
    print(f'indices: {indices}')
    print(f'gemmi_val {gemmi_val}')
    print(f'mrcfile_val {mrcfile_val}')

Output:

gemmi arr: dtype=float32
mrcfile arr: dtype=float32
Not equal indices: (array([ 1,  2,  2, ..., 61, 61, 62]), array([31, 24, 24, ..., 38, 38, 31]), array([32, 29, 30, ..., 33, 34, 31]))
indices: (62, 31, 31)
gemmi_val 0.09139659255743027
mrcfile_val 0.0
indices: (1, 31, 32)
gemmi_val 0.0

Best regards,
Aliaksei

Opening a new mmap with an extended header is slow

Hi Colin

At the moment, when you open a new mmap and want to set the extended header, you first have to rewrite the whole file which if the file is large can take a long time. I found a work around for this but it is pretty nasty!...

        shape = (45, 4000, 4000)

        # Open the handle to the mrcfile
        handle = mrcfile.new_mmap(
            filename,
            shape=(0, 0, 0),
            mrc_mode=mrcfile.utils.mode_from_dtype(dtype),
            overwrite=True,
        )

        # Setup the extended header
        extended_header = np.zeros(shape=shape[0], dtype=mrcfile.dtypes.FEI1_EXTENDED_HEADER_DTYPE)

        # Set the extended header
        handle._check_writeable()
        handle._close_data()
        handle._extended_header = extended_header
        handle.header.nsymbt = extended_header.nbytes
        handle.header.exttyp = "FEI1"
        handle._open_memmap(dtype, shape)
        handle.update_header_from_data()
        handle.flush()

Essentially I create a file with data size 0 and then set the extended header and resize the file. It would be better if you could set the extended header in the constructor perhaps instead?

Best wishes
James

it'd be handy to have validator available from the command line

For example you could throw in __main__.py file:

import sys
from .__init__ import validate

sys.exit(0 if all(validate(path) for path in sys.argv[1:]) else 1)

and then one could validate a map as:

$ python -m mrcfile my.map

writing voxels sizes

Hi Colin,
I am trying to write a volume to disk using mrcfile and setting the header information in the new volume from an existing one by setting directly to the header attribute. This does not appear to be supported but there is no convenient way to serialise/deserialise the rec array that I can see. Would consider adding support for this use case? Alternatively, could you recommend a way to achieve this?

Errors when running against numpy 1.14

Errors come from deprecation of numpy.fromstring() in 1.14. The suggested replacement is to use numpy.frombuffer() but this causes other errors. Needs more investigation. (Note this doesn't affect numpy.rec.fromstring().)

Origin in header is read-only

Hey @colinpalmer !

I'm trying to write a file with a modified origin and am running into

  File "", line 42, in main
    mrc.header.origin = origin_ang
  File "", line 506, in __setattr__
    return self.setfield(val, *res)
ValueError: assignment destination is read-only

I'm getting the same when I try to assign to the x, y or z fields individually. Is this intentional? Any chance we can make this writable?

Many thanks!

Alister

Permissive validation causes memory errors

when using validate on a file that is not an mrcfile, being permissive by default can lead to problematic memory allocations; in my case, a file that produced a number of warnings eventually attempted to allocate a 6TB(!) bytearray which would of course cause an error.

as mitigation, i caught the warnings as exceptions and counted that as a False result, but this would leave the file handler open, which could corrupt data. I can skip validator in my code and go straight to:

try:
  with load_functions.open(filepath, permissive=False) as mrc:
    return mrc.validate(print_file=print_file)
except ValueError:
    return False

... but thought it might be useful to allow validate() to set the permissiveness level; PR #32

Happy to make any alterations if this is useful, or stick to the initial validate avoidance strategy, or whatever is best. My first pass at docstring text could probably use a review.

context: Galaxy sniffs input data to determine it's type; mrcfile is one, and the sniffer currently calls validate(). my guess is validator is hard-coded to permissive=True because it's intended for a file you strongly suspect to be valid, not any possible file (as Galaxy uses it).

Add helper functions for handling header labels

It would be useful to have functions to make it easy to work with the labels in MRC file headers. At the moment, users would need to check the nlabl and labels fields themselves, but this is a bit awkward so functions to take care of getting and setting the labels would be helpful.

There is a bit of complexity in adding such functions though. Some questions to consider:

Should we assume nlabl might be set wrongly and check for "text-like" content in the label slot before writing to it?
What character set should we allow for labels? 7-bit printable ASCII characters only is probably the safest option.

module 'mrcfile' has no attribute 'read'

Dear all,

I tried to use .read() and .write() methods of mrcfile, but both outputs error:

AttributeError: module 'mrcfile' has no attribute 'read'

What could possibly be wrong?
I am running it with Python 3.9.0 and mrcfile is 1.3.0.

Best regards,
Aliaksei

AttributeError: NoneType object has no attribute Mrc

While running the below code, I am getting the following error. Can you tell me where I am going wrong?
CODE :
import numpy
import Mrc
a = Mrc.bindFile('/home/smitha/deep-image-prior/data/Falcon_2015_05_14-20_42_18.mrc')
hist = numpy.histogram(a, bins=200)
a.Mrc.info()
wavelength0_nm = a.Mrc.hdr.wave[0]
ERROR:
AttributeError Traceback (most recent call last)
in ()
3 a = Mrc.bindFile('/home/smitha/deep-image-prior/data/Falcon_2015_05_14-20_42_18.mrc')
4 hist = numpy.histogram(a, bins=200)
----> 5 a.Mrc.info()
6 wavelength0_nm = a.Mrc.hdr.wave[0]
7

AttributeError: 'NoneType' object has no attribute 'Mrc'

Unable to fix mrc file header

Hi! I've been trying to fix the header of an mrc file, but the example in the docs doesn't seem to be working for me.

I know you can just ignore this type of error using the permissive=True keyword argument, but I wanted to fix the file properly, since it was causing a bit of confusion.

Details

Operating system: M1 Mac Monteray 12.3.1 and also tried on Linux Ubuntu 20.04 LTS
python version: 3.9
mrcfile version: 1.4.2

Data

I've been working with this example data, untarred with tar -xzvf examples.tar.gz
This data was a segmentation exported from Amira, which apparently doesn't produce a very nice file header in the mrc files.

To reproduce

I used this example from the mrcfile documentaion

First, open the mrcfile and observe the error

import mrcfile
with mrcfile.open(filename) as mrc:
    print(mrc.data)
# ValueError: Map ID string not found - not an MRC file, or file is corrupt

Then, attempt the header fix suggested in the mrcfile docs

with mrcfile.open(filename, mode='r+', permissive=True) as mrc:
    mrc.header.map = mrcfile.constants.MAP_ID

Try opening the (hopefully fixed) file again

with mrcfile.open(filename) as mrc:
    print(mrc.data)
#  ValueError: Unrecognised machine stamp: 0x90 0x42 0x4d 0xc3

I also tried this suggestion in the comment

import mrcfile
with mrcfile.open('<name>', mode='r+', permissive=True) as mrc:
    mrc.update_header_from_data()

This second option did not seem to update the header at all, because when I tried to read the file back in again, I got the same VAlueError about the Map ID string as I did originally.

Related discussions

What is the license of the official spec?

I wonder if you are going to license it under public domain to make copying the parts of the spec (the docs) into source code legal.

[Feature request] Ability to read from IOStream object directly

Hi Team,
the mrcfile library assumes the source file is coming from a local storage (which IMHO is an incorrect assumption in 2023 ;)). Would you consider changing the API so the IOStream object is allowed to be passed directly to the MrcFile "constructor"?
Cheers,
Irek

Error when opening MotionCor2 output

Hi developers,
Thanks for this useful package. I met a problem when using it to open a single frame data from MotionCor2. Here is the error.
In [9]: mrcfile.validate('20200713avg.mrc')
/software/anaconda/lib/python3.7/site-packages/mrcfile/mrcinterpreter.py:208: RuntimeWarning: Map ID string not found - not an MRC file, or file is corrupt warnings.warn(msg, RuntimeWarning) Map ID string is incorrect: found b'MAP', should be b'MAP ' Error in data statistics: RMS deviation is 31.696720123291016 but the value in the header is 0.0
Out[9]: False
Thanks for your help.

commands do not support a --help option

None of

mrcfile-header -h
mrcfile-header --help
mrcfile-validate -h
mrcfile-validate --help

provide any help as to how the commands are supposed to be used.

Does mrcfile read maps using data dtype according to ccp4 mode when memory mapping is used?

Dear all,

Could you tell me please if mrcfile read map with data dtype according to ccp4 mode, when memory mapping is used?
E.g., if I do:

with mrcfile.mmap(map_file) as mrc_original:
        data: np.memmap = mrc_original

where map_file is map with certain mode, will the data have the dtype corresponding to that mode?

Add option for mrcfile-header to print extended header when it has a known type

Having trouble while saving the image into .TIF

Hi there,
First of all, thank you for this wonderful package in the python environment.
I have downloaded the MRC file from here: https://www.ebi.ac.uk/empiar/EMPIAR-10324/ and was trying to save each slice in a .tif format. The code run very well but the saved image is not very good, you can see the generated image below.
https://uapt33090-my.sharepoint.com/:f:/g/personal/ygupta_ua_pt/Eh0yGt6IzTZOh703l3KY0JcB2zAG46W_TaemNLJNWNk2Dw?e=OJTUtQ
code:

i = mrcfile.open(input1)
for ii in range(i.data.shape[0]):
    ii = 43
    tifffile.imwrite(str(output) + f'{ii}.tif', i.data[ii])

But when I try to save the MRC image into .tif utilizing the IMOD LINUX package it works perfectly.
Can you help me why I am not getting satisfactory images in python?
Thank you!

Test failures with Python 3.11

There's a bunch of test failures with Python 3.11.0b3, apparently due to exception messages having changed:

======================================================================
FAIL: test_setting_data_attribute_raises_exception (tests.test_bzip2mrcfile.Bzip2MrcFileTest.test_setting_data_attribute_raises_exception)
----------------------------------------------------------------------
AttributeError: property 'data' of 'Bzip2MrcFile' object has no setter

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 102, in test_setting_data_attribute_raises_exception
    with self.assertRaisesRegex(AttributeError, "can't set attribute"):
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: "can't set attribute" does not match "property 'data' of 'Bzip2MrcFile' object has no setter"

======================================================================
FAIL: test_setting_extended_header_attribute_raises_exception (tests.test_bzip2mrcfile.Bzip2MrcFileTest.test_setting_extended_header_attribute_raises_exception)
----------------------------------------------------------------------
AttributeError: property 'extended_header' of 'Bzip2MrcFile' object has no setter

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 98, in test_setting_extended_header_attribute_raises_exception
    with self.assertRaisesRegex(AttributeError, "can't set attribute"):
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: "can't set attribute" does not match "property 'extended_header' of 'Bzip2MrcFile' object has no setter"

======================================================================
FAIL: test_setting_header_attribute_raises_exception (tests.test_bzip2mrcfile.Bzip2MrcFileTest.test_setting_header_attribute_raises_exception)
----------------------------------------------------------------------
AttributeError: property 'header' of 'Bzip2MrcFile' object has no setter

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 94, in test_setting_header_attribute_raises_exception
    with self.assertRaisesRegex(AttributeError, "can't set attribute"):
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: "can't set attribute" does not match "property 'header' of 'Bzip2MrcFile' object has no setter"

======================================================================
FAIL: test_setting_data_attribute_raises_exception (tests.test_gzipmrcfile.GzipMrcFileTest.test_setting_data_attribute_raises_exception)
----------------------------------------------------------------------
AttributeError: property 'data' of 'GzipMrcFile' object has no setter

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 102, in test_setting_data_attribute_raises_exception
    with self.assertRaisesRegex(AttributeError, "can't set attribute"):
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: "can't set attribute" does not match "property 'data' of 'GzipMrcFile' object has no setter"

======================================================================
FAIL: test_setting_extended_header_attribute_raises_exception (tests.test_gzipmrcfile.GzipMrcFileTest.test_setting_extended_header_attribute_raises_exception)
----------------------------------------------------------------------
AttributeError: property 'extended_header' of 'GzipMrcFile' object has no setter

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 98, in test_setting_extended_header_attribute_raises_exception
    with self.assertRaisesRegex(AttributeError, "can't set attribute"):
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: "can't set attribute" does not match "property 'extended_header' of 'GzipMrcFile' object has no setter"

======================================================================
FAIL: test_setting_header_attribute_raises_exception (tests.test_gzipmrcfile.GzipMrcFileTest.test_setting_header_attribute_raises_exception)
----------------------------------------------------------------------
AttributeError: property 'header' of 'GzipMrcFile' object has no setter

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 94, in test_setting_header_attribute_raises_exception
    with self.assertRaisesRegex(AttributeError, "can't set attribute"):
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: "can't set attribute" does not match "property 'header' of 'GzipMrcFile' object has no setter"

======================================================================
FAIL: test_setting_data_attribute_raises_exception (tests.test_mrcobject.MrcObjectTest.test_setting_data_attribute_raises_exception)
----------------------------------------------------------------------
AttributeError: property 'data' of 'MrcObject' object has no setter

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 102, in test_setting_data_attribute_raises_exception
    with self.assertRaisesRegex(AttributeError, "can't set attribute"):
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: "can't set attribute" does not match "property 'data' of 'MrcObject' object has no setter"

======================================================================
FAIL: test_setting_extended_header_attribute_raises_exception (tests.test_mrcobject.MrcObjectTest.test_setting_extended_header_attribute_raises_exception)
----------------------------------------------------------------------
AttributeError: property 'extended_header' of 'MrcObject' object has no setter

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 98, in test_setting_extended_header_attribute_raises_exception
    with self.assertRaisesRegex(AttributeError, "can't set attribute"):
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: "can't set attribute" does not match "property 'extended_header' of 'MrcObject' object has no setter"

======================================================================
FAIL: test_setting_header_attribute_raises_exception (tests.test_mrcobject.MrcObjectTest.test_setting_header_attribute_raises_exception)
----------------------------------------------------------------------
AttributeError: property 'header' of 'MrcObject' object has no setter

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 94, in test_setting_header_attribute_raises_exception
    with self.assertRaisesRegex(AttributeError, "can't set attribute"):
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: "can't set attribute" does not match "property 'header' of 'MrcObject' object has no setter"

======================================================================
FAIL: test_setting_data_attribute_raises_exception (tests.test_mrcinterpreter.MrcInterpreterTest.test_setting_data_attribute_raises_exception)
----------------------------------------------------------------------
AttributeError: property 'data' of 'MrcInterpreter' object has no setter

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 102, in test_setting_data_attribute_raises_exception
    with self.assertRaisesRegex(AttributeError, "can't set attribute"):
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: "can't set attribute" does not match "property 'data' of 'MrcInterpreter' object has no setter"

======================================================================
FAIL: test_setting_extended_header_attribute_raises_exception (tests.test_mrcinterpreter.MrcInterpreterTest.test_setting_extended_header_attribute_raises_exception)
----------------------------------------------------------------------
AttributeError: property 'extended_header' of 'MrcInterpreter' object has no setter

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 98, in test_setting_extended_header_attribute_raises_exception
    with self.assertRaisesRegex(AttributeError, "can't set attribute"):
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: "can't set attribute" does not match "property 'extended_header' of 'MrcInterpreter' object has no setter"

======================================================================
FAIL: test_setting_header_attribute_raises_exception (tests.test_mrcinterpreter.MrcInterpreterTest.test_setting_header_attribute_raises_exception)
----------------------------------------------------------------------
AttributeError: property 'header' of 'MrcInterpreter' object has no setter

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 94, in test_setting_header_attribute_raises_exception
    with self.assertRaisesRegex(AttributeError, "can't set attribute"):
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: "can't set attribute" does not match "property 'header' of 'MrcInterpreter' object has no setter"

======================================================================
FAIL: test_setting_data_attribute_raises_exception (tests.test_mrcfile.MrcFileTest.test_setting_data_attribute_raises_exception)
----------------------------------------------------------------------
AttributeError: property 'data' of 'MrcFile' object has no setter

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 102, in test_setting_data_attribute_raises_exception
    with self.assertRaisesRegex(AttributeError, "can't set attribute"):
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: "can't set attribute" does not match "property 'data' of 'MrcFile' object has no setter"

======================================================================
FAIL: test_setting_extended_header_attribute_raises_exception (tests.test_mrcfile.MrcFileTest.test_setting_extended_header_attribute_raises_exception)
----------------------------------------------------------------------
AttributeError: property 'extended_header' of 'MrcFile' object has no setter

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 98, in test_setting_extended_header_attribute_raises_exception
    with self.assertRaisesRegex(AttributeError, "can't set attribute"):
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: "can't set attribute" does not match "property 'extended_header' of 'MrcFile' object has no setter"

======================================================================
FAIL: test_setting_header_attribute_raises_exception (tests.test_mrcfile.MrcFileTest.test_setting_header_attribute_raises_exception)
----------------------------------------------------------------------
AttributeError: property 'header' of 'MrcFile' object has no setter

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 94, in test_setting_header_attribute_raises_exception
    with self.assertRaisesRegex(AttributeError, "can't set attribute"):
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: "can't set attribute" does not match "property 'header' of 'MrcFile' object has no setter"

======================================================================
FAIL: test_setting_data_attribute_raises_exception (tests.test_mrcmemmap.MrcMemmapTest.test_setting_data_attribute_raises_exception)
----------------------------------------------------------------------
AttributeError: property 'data' of 'MrcMemmap' object has no setter

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 102, in test_setting_data_attribute_raises_exception
    with self.assertRaisesRegex(AttributeError, "can't set attribute"):
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: "can't set attribute" does not match "property 'data' of 'MrcMemmap' object has no setter"

======================================================================
FAIL: test_setting_extended_header_attribute_raises_exception (tests.test_mrcmemmap.MrcMemmapTest.test_setting_extended_header_attribute_raises_exception)
----------------------------------------------------------------------
AttributeError: property 'extended_header' of 'MrcMemmap' object has no setter

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 98, in test_setting_extended_header_attribute_raises_exception
    with self.assertRaisesRegex(AttributeError, "can't set attribute"):
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: "can't set attribute" does not match "property 'extended_header' of 'MrcMemmap' object has no setter"

======================================================================
FAIL: test_setting_header_attribute_raises_exception (tests.test_mrcmemmap.MrcMemmapTest.test_setting_header_attribute_raises_exception)
----------------------------------------------------------------------
AttributeError: property 'header' of 'MrcMemmap' object has no setter

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/mrcfile/tests/test_mrcobject.py", line 94, in test_setting_header_attribute_raises_exception
    with self.assertRaisesRegex(AttributeError, "can't set attribute"):
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: "can't set attribute" does not match "property 'header' of 'MrcMemmap' object has no setter"

----------------------------------------------------------------------
Ran 730 tests in 14.119s

FAILED (failures=18)

ValueError: zero-size array to reduction operation minimum which has no identity

value error when I use set_data() on a mrc file

Traceback (most recent call last):
  File "test_crop_img.py", line 31, in crop_mrc
    mrc_crop.set_data(mrc.data[z:dz, x:dx, y:dy])  
  File "/ldap_shared/home/v_wending_peng/anaconda3/lib/python3.7/site-packages/mrcfile/mrcobject.py", line 226, in set_data
    self.update_header_stats()
  File "/ldap_shared/home/v_wending_peng/anaconda3/lib/python3.7/site-packages/mrcfile/mrcobject.py", line 446, in update_header_stats
    self.header.dmin = self.data.min()
  File "/ldap_shared/home/v_wending_peng/anaconda3/lib/python3.7/site-packages/numpy/core/_methods.py", line 32, in _amin
    return umr_minimum(a, axis, None, out, keepdims, initial)
ValueError: zero-size array to reduction operation minimum which has no identity

Any reason for a flip in y

We are seeing a weird behaviour, or maybe not, when Cryolo, that uses this module, reads a MRC file that , wronlgy has 1 in the ispg word being a 2d image not a volume: image is visualize flipped in y. Any reason?

Asynchronous background loading of MRC files

A few people have requested the ability to open MRC files in the background, so they can load one file while processing another.

I've tried a few different approaches now, using Python's threading and multiprocessing modules.

To test the speed of the different options, I've tested opening 6 MRC files (sizes between 55 MB and 1.1 GB) and summing all of their data values. The code for the serial version of the test looks like this:

total = 0
for name in file_list:
    mrc = mrcfile.open(name)
    total += mrc.data.sum()

The tests were all done on Linux. Running times are:

Method	Python 2.7	Python 3.6
Serial opening using normal `mrcfile.open` function	3.4 s	3.7 s
Parallel opening using a new thread for each file	2.3 s	2.2 s
Parallel opening using a thread pool	2.4 s	2.7 s
Parallel opening using a process pool	12.2 s	Fails with `MaybeEncodingError`, `Reason: 'TypeError("cannot serialize '_io.BufferedReader' object",)'`

These results agree with the idea (found in discussions of Python's GIL online) that parallelisation with Python processes is useful for CPU-bound tasks, but thread-based parallelisation is better for I/O-bound tasks. Process-based parallelisation requires objects to be serialised for passing between processes, which introduces a big overhead when handling large arrays of data. Thread-based parallelisation allows the objects to be shared between threads with no copying or serialisation needed.

Therefore, I'm going to go ahead and implement an asynchronous loading function using a simple new-thread-for-each-file approach. (Ideally it would probably be better to use a ThreadPoolExecutor from Python's concurrent.futures module, but that was only introduced in Python 3.2 so doesn't work in Python 2.7. A thread pool is available in Python 2.7 but it's undocumented and seems not worth the effort of using at this point.)

accessing specific slice from a memmap

Hi,

we are often dealing with tiled montage maps of grid squares stored as a stack of montages/mosaics in a single MRC file. These can than easily go into several GB of file size.

I now like to extract only single slices for processing and was looking into the memmaps to do this efficiently.

I tried
mm._open_memmap(dtype='int16',shape=(mm.header.nx,mm.header.ny,mm.header.nz))
giving me a 'NoneType' object back.

How do I access data blocks properly? I could get it directly with _read_data but to get there I would need to specify the exact byte offset. What is the proper wrapper to do it?

Thanks a lot,
Martin

ValueError: cannot set WRITEABLE flag to True of this array

Hi,
When I use mrc.open(file,'r') to open a mrcfile, sometimes the it raises ValueError: cannot set WRITEABLE flag to True of this array.
I don't know how to fix it by change parameters. I add # in mrcfile/mrcfileinterpreter.py 141 line header.flags.writeable = True, and make it run. How could I fix it in a better way?
Thanks.

Proposal: `mrcfile.read()` and `mrcfile.write()`

Hey Colin,

Merry Christmas! 🎄

In the spirit of 'simple is better than complex', would you be open to a PR adding simple read and write functions to mrcfile?

I know this isn't difficult to achieve with the current API but I find myself reaching for/reimplementing these as convenience functions often enough that I thought I'd see what you think

read would probably look along the lines of...

def read(filename):
    with mrcfile.open(filename) as mrc:
        data = np.copy(mrc.data)
    return data

and write...

def write(data, filename, overwrite=False):
    # set `mode` depending on value of overwrite
    with mrcfile.new(filename, mode) as mrc:
        mrc.set_data(data)

ccpem / mrcfile Goto Github PK

mrcfile's Introduction

mrcfile.py

Key Features

Installation

Basic usage

Documentation

Citing mrcfile

Contributing

Licence

mrcfile's People

Contributors

Stargazers

Watchers

Forkers

mrcfile's Issues

Details

Data

To reproduce

Related discussions

Recommend Projects

Recommend Topics

Recommend Org