nion-software / niondata Goto Github PK

This mechanism could be used during acquisition and after canceled long operations. Processing functions could apply only to the valid portion.

The current implementation uses the default data_descriptor from DataAndMetadata for the result of concatenate. This can lead to unexpected result when concatenating e.g. two sequences, because the result would then be a collection (which is the default behavior of DataAndMetadata).
To decide which data_descriptor to use there are two options:

1. Use the data_descriptor of the first input xdata. This would match the current behavior for intensity_calibration

2. Only allow concatenation of xdata with homogeneous data_descriptors and use the common data_descriptor.

The second method would be "cleaner" but is also more restrictive and could cause some confusion for users. On the other hand, concatenating non-homogeneous data is not a well-defined operation so it should maybe not be allowed.

'reshape', for instance

Add tests.

The DataAndMetadata object includes the ability to describe the data type, dimensional usage, and calibrations. But there is no mechanism to track the overall nature of the data array such as whether complex data is Hermitian.

See nion-software/nionswift#242

The Hermitian-ness of the data can be determined programmatically, but it would be time consuming. So having some mechanism of tracking it would be useful.

Also, it might be nice to treat Hermitian-ness as a data type (i.e. complex64-hermitian) but this might be difficult to support when transparently masquerading as an ndarray since ndarray has no such data type.

Storing this as a property in metadata would be another possibility. But two things make this difficult: the first is the abstract design idea that the metadata is not descriptive of the data mechanics; the second is that metadata is stripped during any processing (this behavior is currently being debated; but that's how it works right now).

That leaves adding another descriptive element to DataAndMetadata to describe whether complex data is to be considered Hermitian.

Are there other similar descriptive possibilities other than Hermitian?

Multiplying a calibrated image by an uncalibrated image should result in a calibrated image.

• #16

BPS email 2021-09-01

Right now, some of them at least try to add some sensible dimensional and intensity calibrations to the output data. But all functions in Core.py strip the metadata dict from their inputs.
Processing functions with one input and one output should just copy the metdata to their outputs.
For processing functions with multiple inputs, I'm not sure what the best solution would be. 3 ideas:

1. Use the metadata from the first input: Very simple but probably the worst solution (still better than the current situation, though)
2. Add a list of metadata dicts to the output: Also very simple, but can overload the metadata of the output a bit. We already have a lot of stuff in there, and multiplying that might make it even harder for the user to browse through it.
3. Merge the metadata dicts: This is the most complex, but probably most elegant solution. We would recursively go through the metadata and only put common keys into the result once. If the values of these keys are the same, we just list the values once, otherwise they get turned into a list of values for each input.
   Alternatively we could have the common key, value pairs in one dict and the "unique" ones in a list of dicts as in point 2.

I have an issue related to sequences of color images. A minimal reproducible example, that can be typed into the console, is given below.

data = (numpy.random.rand(4, 64, 64, 3) * 255).astype(np.uint8)
data_descriptor = api.create_data_descriptor(is_sequence=True, collection_dimension_count=0, datum_dimension_count=2)
xdata = api.create_data_and_metadata(data, data_descriptor=data_descriptor)
api.library.create_data_item_from_data_and_metadata(xdata)

—— error message (shortened) ——

dimensional_calibration = data_and_metadata.dimensional_calibrations[dimensional_calibration_index]

IndexError: list index out of range

A simple change to the function “function_data_slice” in nion.data.DataAndMetadata.py makes color sequences work. However, this change currently messes up 1d data.