Opening partly written HDF5 files with happi after simulation crash about smilei HOT 16 CLOSED

Thank you for suggesting this. I actually had the same comment a few days ago from a colleague.

from smilei.

ok, sent again, hopefully right person this time :)

from smilei.

Admittedly the 'flush_every' function helps reduce the chance of the HDF5 file being corrupted during a write while it's running, so part of what I said is not quite right, but the problem still exists that some bits of data at the end of the HDF5 files is miswritten sometimes during a crash.

from smilei.

Sorry, bad code in previous thing, this recreates the probes hdf5 file (at least I think it does) and seems to work all of the time as far as I have tested:

`

import h5py
f_dest = h5py.File("Probes0_fixed.h5", "w")
f_src = h5py.File("Probes0.h5", "r")
for key in f_src:
    try:
        f_dest.create_dataset_like(str(key), f_src[key])
        f_dest[key][()] = f_src[key][()]
        for attrib in f_src[key].attrs.keys():
            f_dest[key].attrs.create(attrib, f_src[key].attrs[attrib])

    except KeyError:
        print("faulty key = " + str(key))

for attrib in f_src.attrs.keys():
    f_dest.attrs.create(attrib, f_src.attrs[attrib])

`

from smilei.

Just to note when I use the above script it doesn't always work. The individual attributes must also be tested before the key is written to the new H5 file as sometimes a key can be correctly created but not filled with attributes correctly.

from smilei.

Do you have an idea of to reproduce this? I cannot get a corrupted file

from smilei.

from smilei.

It does not happen on my system for some reason. Would you be able to produce a small example and send it with dropbox or equivalent?

from smilei.

from smilei.

It does not happen on my system for some reason. Would you be able to produce a small example and send it with dropbox or equivalent?

I've sent you a link in element chat

from smilei.

I have not received it. My name on element is fredpz

from smilei.

from smilei.

from smilei.

I made a change for happi in the develop branch. Could you test it?

from smilei.

Yes, that seems to allow me to access files which I couldn't before, thanks! That's eliminated a step which was quite annoying and will save me some time too, really appreciated!

from smilei.

Just to be clear, with the old version I tried to access a broken probes0.h5 file and got this error:

Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/usr/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar
return list(map(*args))
File "/media/david_blackman/left_external/broadband/simulation_results/bandwidth/redone_diags/fixed_ions/narrow/../../../../py1D/make_signal_files.py", line 193, in start
signal = Ez_probe.getData()
File "/home/david_blackman/codes/smilei_new/Smilei/happi/_Diagnostics/Diagnostic.py", line 163, ingetData
data.append( self._dataAtTime(t) )
File "/home/david_blackman/codes/smilei_new/Smilei/happi/_Diagnostics/Diagnostic.py", line 862, in_dataLinAtTime
A = self._getDataAtTime(t)
File "/home/david_blackman/codes/smilei_new/Smilei/happi/_Diagnostics/Probe.py", line 372, in _getDataAtTime
data = self._dataForTime[t][n,first:last]
TypeError: 'NoneType' object is not subscriptable

Now I get no error and successfully build up my probe signals so I can process them properly.!

from smilei.