We have a script to compute the test data used and commited to the repository. It runs in a few seconds. We should consider recreating the data for the tests because we do not detect changes in the code that creates the data. Therefore, we initially missed the recent change in the HysteresisDriver (renaming the hysteresis field) that should have broken the tests in micromagnetictests.

ubermag/mumax3c#35

The implementation of data.info is quite fragile and easily breaks, e.g. if there are missing drives or the info.json of one of the drives cannot be read.

Dear ubermag team，
I need to add current density related parameters, but I don't know how to write instructions. Please tell me how to solve this problem.

Mumax3 shifts pmin to the origin when writing output files. Micromagneticdata tries to load subregions from the json file that is created when writing the input files. This fails if pmin in the input is not at (0, 0, 0).

When accessing data of a drive drive[<number>] we currently use an index based on the available files on disk. We should consider using data from drive.table.data instead. That could be useful when one wants to keep only a few selected magnetisation configurations and the scalar data.

Currently, Data takes two arguments: name and dirname, both have to be strings. It would be more convenient to:

1. convert this or additionally allow a single argument, the full (relative) path to the data folder, i.e. dirname/name
2. support pathlib.Path in addition to str

Check line endings for the following files when running a simulation on Windows:

• OOMMF odt
• Mumax3 ovf
• Mumax3 txt

Update .gitignores accordingly (in micromagneticdata and ubermag).

Loading drive 0, magnetisation snapshot 4 on Windows results in wrong numbers (looks like overflow/encoding problems). Linux and Mac are not affected.

We currently support concatenating tables using the << syntax.

Similar functionality would be very useful for drive objects, e.g. to concatenate two consecutive calls to TimeDriver with drive1 << drive2 and create an interactive plot.

However, there are a few methods/properties that are very specific to one drive:

• m0
• info (a small json file with some information about that drive, e.g. date)
• mif (returns the content of the mif file)

Operating on the drive objects would be good as these are memory efficient (e.g. do not load all data) and I plan to add holoviews based interactive plotting functionality to Drive. Furthermore, the drive object has a reference to the corresponding table object with all the scalar data.

An alternative would be to try to delegate this to xarray. The obvious downside is that this would immediately load all data to memory (probably okay for our typical file sizes). Also, we should then have a different set of plotting functions that operate on the xarray instead of the Drive object. And we would loose the direct link to the table object.

The ovf rectangle drive[0][4] is not read properly on Windows (Linux and Mac are fine), the full error message when running pytest is shown further down. Other ovf files in the output data (same drive) do not show the problem. Re-computing the test data (including changing the system name to get rid of any potential issues in the git history) did not fix the issue.

@samjrholt and I looked at the problematic file on Windows and Mac using a HEX editor (also comparing with a different file from the same drive on Windows that did not have the problem). It appears that in the problematic file on Windows all line endings are changed from LF ('\n') -> LFCR ('\n\r'). It is not clear why:

1. this happens for the binary file (OOMMF always writes just LF, independent of the OS)
2. why the same does not happen for the other ovf files.

We only looked at the ASCII header of the ovf file in detail and did not check for changes in the actual data. We know that the data read from the file is wrong but not whether this is a cause of the broken header or whether the binary part is also modified.

When re-computing the data (running python micromagneticdata/tests/generate_test_data.py) on Windows the file is written correctly (just LF). After committing, pushing and pulling in a fresh clone the data is wrong (on Windows, the same commit is fine on Mac/Linux). We did not try deactivating the automatic LF <-> LFCR conversion in git (such a change should not be required!).

_________________________ TestDrive.test_getitem_int __________________________

self = <micromagneticdata.tests.test_drive.TestDrive object at 0x0000026A70514F70>

    def test_getitem_int(self):
        for i in range(self.data[0].n):
>           assert isinstance(self.data[0][i], df.Field)

i          = 4
self       = <micromagneticdata.tests.test_drive.TestDrive object at 0x0000026A70514F70>

micromagneticdata\tests\test_drive.py:97: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
micromagneticdata\drive.py:212: in __getitem__
    return super().__getitem__(item)
        __class__  = <class 'micromagneticdata.drive.Drive'>
        item       = 4
        self       = OOMMFDrive(name='rectangle', number=0, dirname='D:\a\micromagneticdata\micromagneticdata\micromagneticdata\tests\test_sample', x='t')
micromagneticdata\abstract_drive.py:208: in __getitem__
    field.valid = "norm"
        field      = Field(Mesh(Region(pmin=[-5e-08, -2.5e-08, 0.0], pmax=[5e-08, 2.5e-08, 2e-08], dims=['x', 'y', 'z'], units=['m', 'm', '....0], pmax=[5e-08, 2.5e-08, 2e-08], dims=['x', 'y', 'z'], units=['m', 'm', 'm']))), nvdim=3, vdims: (x, y, z), unit=A/m)
        item       = 4
        self       = OOMMFDrive(name='rectangle', number=0, dirname='D:\a\micromagneticdata\micromagneticdata\micromagneticdata\tests\test_sample', x='t')
C:\Miniconda\envs\conda-environment\lib\site-packages\discretisedfield\field.py:561: in valid
    valid = ~np.isclose(self.norm.array, 0)
        self       = Field(Mesh(Region(pmin=[-5e-08, -2.5e-08, 0.0], pmax=[5e-08, 2.5e-08, 2e-08], dims=['x', 'y', 'z'], units=['m', 'm', '....0], pmax=[5e-08, 2.5e-08, 2e-08], dims=['x', 'y', 'z'], units=['m', 'm', 'm']))), nvdim=3, vdims: (x, y, z), unit=A/m)
        valid      = 'norm'
C:\Miniconda\envs\conda-environment\lib\site-packages\discretisedfield\field.py:527: in norm
    res = np.linalg.norm(self.array, axis=-1, keepdims=True)
        self       = Field(Mesh(Region(pmin=[-5e-08, -2.5e-08, 0.0], pmax=[5e-08, 2.5e-08, 2e-08], dims=['x', 'y', 'z'], units=['m', 'm', '....0], pmax=[5e-08, 2.5e-08, 2e-08], dims=['x', 'y', 'z'], units=['m', 'm', 'm']))), nvdim=3, vdims: (x, y, z), unit=A/m)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

x = array([[[[-6.43428711e+249, -6.31358452e+206,  6.95375381e-025],
         [ 6.95375381e-025, -6.45299271e+009,  4.1692...45299271e+009,  4.16922004e+264, -3.54059361e+186],
         [-3.54059361e+186,  3.01554046e-260, -1.66647543e-246]]]])
ord = None, axis = (-1,), keepdims = True

    @array_function_dispatch(_norm_dispatcher)
    def norm(x, ord=None, axis=None, keepdims=False):
        ...
                # special case for speedup
>               s = (x.conj() * x).real
E               RuntimeWarning: overflow encountered in multiply

axis       = (-1,)
keepdims   = True
nd         = 4
ord        = None
x          = array([[[[-6.43428711e+249, -6.31358452e+206,  6.95375381e-025],
         [ 6.95375381e-025, -6.45299271e+009,  4.1692...45299271e+009,  4.16922004e+264, -3.54059361e+186],
         [-3.54059361e+186,  3.01554046e-260, -1.66647543e-246]]]])

x is the numpy array containing the magnetisation, expected values (i.e. values read on Mac/Linuxr are on the order 1e6.

Numpy does not fail, just shows a warning. Our pytest configuration escalates warnings to errors (see below).

All tests that access the broken file are affected:

=========================== short test summary info ===========================
FAILED micromagneticdata/tests/test_combined_drive.py::TestDrive::test_iter - RuntimeWarning: overflow encountered in multiply
FAILED micromagneticdata/tests/test_combined_drive.py::TestDrive::test_to_xarray - RuntimeWarning: overflow encountered in multiply
FAILED micromagneticdata/tests/test_combined_drive.py::TestDrive::test_register_callback - RuntimeWarning: overflow encountered in multiply
FAILED micromagneticdata/tests/test_drive.py::TestDrive::test_getitem_int - RuntimeWarning: overflow encountered in multiply
FAILED micromagneticdata/tests/test_drive.py::TestDrive::test_getitem_slice - RuntimeWarning: overflow encountered in multiply
FAILED micromagneticdata/tests/test_drive.py::TestDrive::test_iter - RuntimeWarning: overflow encountered in multiply
FAILED micromagneticdata/tests/test_drive.py::TestDrive::test_ovf2vtk - RuntimeWarning: overflow encountered in multiply
FAILED micromagneticdata/tests/test_drive.py::TestDrive::test_to_xarray - RuntimeWarning: overflow encountered in multiply
FAILED micromagneticdata/tests/test_drive.py::TestDrive::test_register_callback - RuntimeWarning: overflow encountered in multiply
FAILED micromagneticdata/tests/test_drive.py::TestDrive::test_cache - RuntimeWarning: overflow encountered in multiply

The problem is probably not new. It showed up now because we changed all warnings to errors (to notice such problems). We decided to for now ignore this particular problem as we do not know how to fix it (re-computing the test data as part of the CI is not an option because some of the simulations require Mumax3, hence a GPU [not available in the CI], to cover a broader range of possible files that we should test for).

Any ideas where the problem could come from would be highly appreciated @fangohr, @swapneelap, @marijanbeg .

Currently, it is not possible to see, if the runs have actually been completed. It would be useful to add an extra column that indicates the final status (success/finished or failure/interrupted) and for time drives (maybe also hysteresis drives) the number of steps that have actually been calculated.