Follow the guide used by Mona to implement timing calls for the MPI routines (exa_slurm/auto/worker.py)

During a recent experiment, a very dense dataset was collected, including many lattices per frame. Usually, during XFEL processing the import, find spots, index, refine and integrate steps take roughly equal amounts of time. Here, however, when we used multiple lattice indexing and indexing.method_list=fft1d real_space_grid_search fft3d, nearly 90% of the processing time was spent in the indexing step, and most of that time is spent in the function find_candidate_orientation_matrices in dials/algorithms/indexing/indexer.py. Specifically, I ran xtc_process.py single process on a small run with 310 events. The program ran for 6100s, of which 5500s was spent in indexing, of which 3000s was spent in find_candidate_orientation_matrices, or about half.

The profile from cProf is available on request.

Currently to install and run psana/cctbx builds the mapping of XTC files to experiment names is contained within the experiment-db.dat. This file must be rsynced from LCLS systems at SLAC, and is not currently available to the general public. Having the file included with the conda installation of psana would allow users to update the file by updating their version of psana.

TODO (lifted from dials/dials#885)

• Replace dials.util.halraiser with dials.util.show_mail_on_error context manager;
• Replace dxtbx.model.experiment_list.ExperimentListDumper and dxtbx.serialize.dump with as_file method of experiment list
• Replace dials.algorithms.indexing.index_reflections with dials.algorithms.indexing.assign_indices.AssignIndicesGlobal (not in xfel but could be in extant repositories)
• Fix DeprecationWarning: Setting verbosity for a Refiner is deprecated. See dials/dials#860

The multiprocess submission code reads the incorrect file for substitution.

```self.srun_contents = self.substitute(self.srun_contents, "<nproc>",
  str(self.params.shifter.nproc))```

should be

self.srun_contents = self.substitute(self.sbatch_contents, "<nproc>", str(self.params.shifter.nproc))