lanl / milk Goto Github PK

Assert that CIF files in MILK.json exist at import

By streaming the refinement progress a user can check whether MAUD is progressing or if there is an error that is making it hang.

If the background loop doesn't exist the individual background cannot be modified by MILK.

Add time stamp in lines

Starting MAUD refinement for step 5

such that user can see if something takes way too long...

It seems that since the task is running outside the terminal, the terminal does not recognize that it is even running a task. The option to terminate task is greyed out. Maybe we should add a stand-alone way of terminating the cinema server.

Need to add the riet_par_background_pol and loop keys to all datasets.

When MILK is called with arg parse, a split by space is used on arguments. This does not work when there is a space in a directory name since it is split up.

There is a bug in the upper d-spacing limit for the 144 detectors - I believe we had this discussed before? It doesn't set the proper limit for the last of three rotations, only the first two.

Previously I thought it is entry point issue. I am still not sure if it is, but I solved it by comparing milk-integrate with milk-esg-loader.

Specifically, in if __name__ == "__main__":

milk-esg-loader is just:

if __name__ == "__main__":
    main()

while milk-integrate is:

if __name__ == "__main__":
    freeze_support()
    args = get_arguments()
    main(files=args.FILE, json_file=args.json, output=args.output,
         overwrite=args.overwrite, poolsize=args.poolsize, formats=args.format, histogram_plot=args.histogram_plot, quiet=args.quiet)

I was able to resolve the argument issue by changing it to:

if __name__ == "__main__":
    freeze_support()
    main()

And in main:

def main():
    """Build integration file set and objects and performed integration.

    Args:
        file (list(str)): Input file(s) for integration. parsed using pathlib glob and can contain wild cards.
        json (str): Configuration file for integration.
        output (str, optional): Directory for files to be saved to. Defaults to None.
        overwrite (bool, optional): Overwrite previous integration results. Defaults to False.
        poolsize (int, optional): Number of processors to use. Defaults to None.
        format (list(str), optional): Export format of integration result. Defaults to 'dat'.
        format (bool, optional): Export png histogram plots . Defaults to 'False'.
        format (bool, optional): Turn off terminal messages. Defaults to 'False'.
    """

    args = get_arguments()
    files=args.FILE
    json_file=args.json
    output=args.output
    overwrite=args.overwrite
    poolsize=args.poolsize
    formats=args.format
    histogram_plot=args.histogram_plot
    quiet=args.quiet

Problem might be related to running only single analysis? Don't see it when running several analyses.

For at least example 4: CHESS-GE, pathos is required but it is removed because Spotlight is currently removed for Windows.

I resolved it by adding Pathos back to pip install section for milk. Everything works.

Please confirm and I can push this.

Not sure where the zfill class parameter is actually applied.

The line
editor.free(key='Biso')
will not refine Biso, the correct key is "B_iso"...

Minimum python version going to be set to 3.9 to support typing interfaces.

If input files in e.g. dataset.json are wrong (path not updated, no files etc.) are wrong, py script just returns. Would be useful to see out put like "Creating analysis run with XXXXX.gda, YYYYY.gda, ZZZZZ.gda" or a "no files found - update dataset.json?"

The issue occurs in cake2MAUD. I am attaching the input files that match the way we did CHESS, APS, and SLAC-MEC. The input calibration files match those on your computer. euXFEL-HED.zip. A local copy of the milk_integrate_local.py and the original milk-integrate-mg.py are included.

The specific issue originates from this part in the current version of the script:

def load_interpolation(fname):
        with open(fname, 'rb') as f:
            shape_stored = pickle.load(f)
            radial_stored = pickle.load(f)
            print(pickle.load(f))
            azimuthal_stored = pickle.load(f)
            X_bin = pickle.load(f)
            Y_bin = pickle.load(f)
        return X_bin, Y_bin, shape_stored, radial_stored, azimuthal_stored

The original milk-integrate-mg.py runs successfully but the current version returns error:

Traceback (most recent call last):
  File "C:\Users\jesse\.conda\envs\rietveld\Scripts\milk-integrate-script.py", line 33, in <module>
    sys.exit(load_entry_point('MILK', 'console_scripts', 'milk-integrate')())
  File "d:\1lanlwork\2023lanl\milk\bin\milk_integrate.py", line 79, in entry_point
    main(files=args.FILE, json_file=args.json, output=args.output,
  File "d:\1lanlwork\2023lanl\milk\bin\milk_integrate.py", line 761, in main
    integrate(detectors, output, overwrite, formats,
  File "d:\1lanlwork\2023lanl\milk\bin\milk_integrate.py", line 602, in integrate
    integration2d(mask, mg, data, opts, stem, formats, histogram_plot)
  File "d:\1lanlwork\2023lanl\milk\bin\milk_integrate.py", line 675, in integration2d
    intensity_det, X_bin_det, Y_bin_det, sigmas_det = cake2MAUD(
  File "d:\1lanlwork\2023lanl\milk\bin\milk_integrate.py", line 304, in cake2MAUD
    X_bin, Y_bin = get_interpolation(
  File "d:\1lanlwork\2023lanl\milk\bin\milk_integrate.py", line 284, in get_interpolation
    X_bin, Y_bin, shape_stored, radial_stored, azimuthal_stored = load_interpolation(
  File "d:\1lanlwork\2023lanl\milk\bin\milk_integrate.py", line 276, in load_interpolation
    azimuthal_stored = pickle.load(f)
EOFError: Ran out of input

I see that the two scripts have different cake2MAUD implementations. But I am having difficulty understanding why the input files created following the same procedure I did for the other detector examples won't work with the current version of milk-integrate.

hippo.json has lines

"omega_meas": [-45.0,22.5,45],
"chi_meas": [0.0,0.0,0.0],
"phi_meas": [0.0,0.0,0.0],
"rot_names": ["omega -45.0","omega 22.5","omega 45.0"],

Initial.par with new cinema-milk has still bank names like "Bank144 omega 90.0", so does Analysis.par after running 1_build_database.py. This leads to

Traceback (most recent call last):
File "2_setup.py", line 123, in
editor = configure_hippo_parameters(editor, config_hippo)
File "2_setup.py", line 66, in configure_hippo_parameters
value=f'{editor.value[0]} 0 100000',
IndexError: list index out of range

which is somewhat cryptic but can be traced back to

# Fix one difc to break lattice parameter correlation
editor.get_val(key='_instrument_bank_difc',
               loopid='0',
               sobj=f'{hippo["bank_prefix"]}{hippo["detectors"][0]} {hippo["rot_names"][0]}',
               nsobj='90.0')

in function
def configure_hippo_parameters(editor, hippo) -> MILK.parameterEditor.editor:
building a search term "Bank144 omega -45.0" based on the hippo.json. Looks like the 1_build_database.py doesn't rename the bank names properly from the template? Can the get_val method print an error message if a search term is not found like in this case, likely abort with a "search term not found, this will lead to unpredictable results, aborting!" or so?

The lines

    if i > 0:
        editor.ref(key1='Biso', key2='Biso', value='0 1 100000', sobj1=phases[0], sobj2=phase)
    else:
        editor.set_val(key='Biso', sobj=phase, value='0.6')

do not work once a CIF file has multiple atoms. Can they be made more general, i.e. loop over all atoms in a structure?

Dataset swap using MILK's swap dataset name runs into problems if there is path to the data.

Need to add a timeout feature or seg fault detection to the MAUD call.

If a negative number is passed as a value key, sparatically argparser will treat it as another key argument instead of a value argument.

Example

Changing the negative number to not be in engineering format seems to work correctly:

Can we hide by default all uncertainties (*_e fields) and less used items like GOF, RwP, cell angles etc.? It takes a minute each time the browser comes up to always hide the same ones...

https://github.com/cctbx/cctbx_project

Should we just include both tutorial.sh and tutorial.bat in the folder and instruct the user to use the appropriate one?

the key _maud_output_diff_data_filename isn't implemented

Simulation can be performed with just a pattern evaluation for given set of detectors and data. The method can be added to the maudText methods and take an editor instance, a user function for setting simulation parameters, and a simulation parameter dictionary dataframe. the extract_spectra function can then be used to load and store data. If parallelism applied over the dataframe, there shouldn't be a need for users to manage the file directory.

Simple example to start from: simulate.pdf

The assert " Your second argument specified multiple lines. Only one line can be referenced at a time!" should also state that this assert gets triggered when no line was found. Ideally it should give the sobj names etc. so the user can debug which phase names etc. might be wrong?

Had the error

Traceback (most recent call last):
File "/home/sven/hippo/23/ginny/2_setup.py", line 133, in
editor.ifile = set_dataset_starting_values(editor,dataset,config_dataset)
File "/home/sven/hippo/23/ginny/2_setup.py", line 89, in set_dataset_starting_values
if not np.isnan(value):
TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

Was caused by reading dataset.csv with both comma and space as separators leading to two extra columns. A message such as "processing dataset.csv for starting values" would give some sort of a hint for the user where to look for the cause?

milk.json has a line

    "section_title": "Zr-2.5Nb full texture",

this leads to an error:

2_setup.py: error: unrecognized arguments: full texture01

Suggest to replace spaces with underscores.

Option no longer does anything. Steps > current are already removed to ensure databases make sense.

The new time stamp like

Starting MAUD refinement for step: 1, at: 19:37:22

is useful but would be even better if it also includes a date

The example hippo.json removes banks that were broken or disabled to to operating HIPPO for a few runs in furnace configuration (12,15,etc.) - they should be all enabled, most data in 2021 had even bank 36.

The example milk.json has the construct for wild [0,2] or so - it seems like automatically all runs are processed and N div 3 run00XX folders with the right data are produced. Can the wild business be removed from the milk.json file, just leave the name of the "runXXX" folder root?

Dan, great work with this. I think I found a bug for the windows setup process.

I could not get the MAUD_PATH env variable to correctly set. I think that the bat_install function in config_maud.py needs to write a ".bat" file instead of a ".sh" file. Also, the syntax to define an environment variable is different depending on whether you are using cmd or PowerShell. for a default anaconda installation, it should setup an Anaconda PowerShell and an Anaconda CMD prompt, so it might be good to add compatibility for both.

The current "set MAUD_PATH={maud_path} works with CMD only. For PowerShell, the syntax in the python code I think needs to be $env:MAUD_PATH={maud_path}. I think you could probably add in an if/then statement to cover both options in the .bat file.

Ah the joys of using windows.. Let me know if you want me to submit a pull request or something.

hope this is clear...

At present the wild can be set in milk.json and dataset.csv. 1_...py, 2_setup.py and 3_...py for at least the HIPPO example don't all use the same source for the wild or even read it from run folders created in step1. This can lead to confusion when debugging issues with all 3 steps and assuming e.g. the True/False flags in dataset.csv are used. Maybe output "Using wild from dataset.csv/milk.json/run folders" to tell the user where the runs to be processed are coming from? Or only have one source like dataset.csv for the template scripts?

hippo.json has line

"omega_meas": [-45.0,22.5,45.0],

but after running steps 1 and 2 the resulting AfterSetup.par files have the first and third angle as +45. Also are

editor.get_val(key='_instrument_bank_difc',
               loopid='0',
               sobj=f'{hippo["bank_prefix"]}{hippo["detectors"][0]} {hippo["rot_names"][0]}',
               nsobj='90.0')

in configure_hippo_parameters hard-coded 90 degree angles that fail when other angles are used?

Had

AssertionError: The length of datasets were not the same!

Resolution was to delete a run??? folder. Maybe the print statements for the assertions can output a bit more information for the user on how to fix the problem, like "this typically happens when..." or "try to delete your run folders"?

step 2_setup.py should reset the background parameters imported from the template file and when fitting histogram scale likely also fit the background parameters, like the "scale and background parameters" in the wizard? If template file is done with longer count time than actual runs analyzed with MILK, not resetting the background leads to sub-optimal background parameters

# Use MAUD to load the phase and refine the intensity scaling
#============================================================#
editor.fix_all()
editor = free_bank_parameters('_inst_inc_spectrum_scale_factor', editor, hippo)
# SV added: reset background and fit also background in this step
editor.set_val(key='Background', value='0')
editor.free(key='Background')  # tied bk parameters are ignored

Need to at least update pathos to 0.3.1 for uncertainty examples to work. Try upgrading Mystic too.

I say "may" because it doesn't look faulty to me. However:

Upon running milk_integrate "calibration_files/CeO2_126.tif" --json "1d.azimint.json" --output "results" --overwrite --poolsize 1 --format "esg" --histogram_plot, it returns error:

Traceback (most recent call last):
  File "C:\Users\jesse\.conda\envs\milk2\Scripts\milk-integrate-script.py", line 33, in <module>
    sys.exit(load_entry_point('MILK', 'console_scripts', 'milk-integrate')())
TypeError: main() missing 2 required positional arguments: 'files' and 'json_file'

I copied milk-integrate to the local CHESS-GE folder and run: python milk_integrate_local.py "calibration_files/CeO2_126.tif" --json "1d.azimint.json" --output "results" --overwrite --poolsize 1 --format "esg" --histogram_plot

It completes without issues.

The main definition is
def main(files, json_file, output=None, overwrite=False, poolsize=None, formats=['dat'], histogram_plot=False, quiet=False):

It seems like the entry_point is not passing the arguments.

If user forgets to change folder with GDA files in dataset.json, e.g. leaves it at

"data_dir": "data/",

with the data in some other folder, executing 1_...py gives no feedback that nothing could be done. At least one run should be found...