Currently, singFEL crashes when run with >1 processes. Probably due to mismatch in mpi libraries at compile and run times.

Original implementation in simS2E works such that a photons.dat is generated only once for all
orientation runs. this speeds up calculations if many patterns have to be read.

• Specify working directory for EMC
• If photons.dat already present, skip reading from diffr_out.h5
• Number of runs should always be 1 or implement a scheduling mechanism.
• consider replacing EMC by dragonfly (openmpi implementation of EMC), but different format

I run the simex with wpg method and had a error. The details of error are following.

"

Starting SIMEX photon-matter interaction.

Previous module: small_size_large_detector
NOT: data
NOT: history
misc
params
version
[u'arrEhor', u'arrEver']
[(u'arrEhor', <HDF5 dataset "arrEhor": shape (101, 101, 550, 2), type "<f4">), (u'arrEver', <HDF5 dataset "arrEver": shape (101, 101, 550, 2), type "<f4">)]

Traceback (most recent call last):
File "PhotonExperimentSimulationTest.py", line 44, in
pxs.run()
File "/afs/desy.de/user/l/liujin/SIMEX/v0.1/lib/python2.7/SimEx/PhotonExperimentSimulation/PhotonExperimentSimulation.py", line 161, in run
self.__photon_interactor.backengine()
File "/afs/desy.de/user/l/liujin/SIMEX/v0.1/lib/python2.7/SimEx/Calculators/XMDYNDemoPhotonMatterInteractor.py", line 190, in backengine
pmi_demo.f_load_pulse( pmi_demo.g_s2e['prop_out'] )
File "/afs/desy.de/user/l/liujin/SIMEX/v0.1/lib/python2.7/pmi_demo.py", line 260, in f_load_pulse
self.g_s2e['pulse']['xFWHM'] = xfp.get( '/misc/xFWHM' ) .value
AttributeError: 'NoneType' object has no attribute 'value'
[liujin@exflst314]~/0609_run/9keV/small_size_large_detector% python PhotonExperimentSimulationTest.py > a.txt
Traceback (most recent call last):
File "PhotonExperimentSimulationTest.py", line 44, in
pxs.run()
File "/afs/desy.de/user/l/liujin/SIMEX/v0.1/lib/python2.7/SimEx/PhotonExperimentSimulation/PhotonExperimentSimulation.py", line 161, in run
self.__photon_interactor.backengine()
File "/afs/desy.de/user/l/liujin/SIMEX/v0.1/lib/python2.7/SimEx/Calculators/XMDYNDemoPhotonMatterInteractor.py", line 190, in backengine
pmi_demo.f_load_pulse( pmi_demo.g_s2e['prop_out'] )
File "/afs/desy.de/user/l/liujin/SIMEX/v0.1/lib/python2.7/pmi_demo.py", line 260, in f_load_pulse
self.g_s2e['pulse']['xFWHM'] = xfp.get( '/misc/xFWHM' ) .value
AttributeError: 'NoneType' object has no attribute 'value'
"

The original code is in the directory test in branch xrt_calculator

Interface the difference map phasing algorithm

How will the interfaces to scattering simulations look like?
Specifically, where does the Simex Platform take over from the scattering simulation and vice versa?

When dealing with Monte Carlo photon sampling of scattering the number of photons and the correct addition of all photons according to their phases is a significant numerical challenge due to the precision required and a significant computational challenge due to the large number of photons required. How will the Simex Platform deal with these challenges?

Recent updates in wpg use numpy.ndarray.tobytes(), which was added in numpy version 1.9.
Our current requirements are numpy1.8.2

• Update requirements
• Test on available platforms -> Will fail on xfl cluster and maxwell

The errors reported from travisCI (see e.g. this log: https://travis-ci.org/eucall-software/simex_platform/jobs/132826264) are not reproduced if build+test on local machine. It seems that some test files are not installed correctly in the build on travisCI, see debug print in lines 3635ff. Someone with more experience with travisCI (@ax3l ?) might be quicker to solve this.

Thanks, Carsten

These items are missing before we can run the entire simS2E toolchain in simex_platform:

• Support all parameters in the simS2E config file and elsewhere (i.e. EMC/DM parameters are located in the respective sources.)
• Finish EMC and DM support

Build a reference manual, e.g. through sphinx.

Develop a strategy how to do parallel backengine execution and implement in Calculators.

Install and test simex_platform on the DESY maxwell cluster. (related to #34 )
Document all necessary fixes in README.md

• Allow specification of sample like sample="2nip.pdb"
• If time, implement interface to query from pdb.org

• Implement a generic solution (possibly on the level of the abstract calculators) for handling cases where input consists of a single .h5 file or a directory containing multiple files.
• Consider using always only one single .h5 files with added hierarchy for multiple runs.

(This issue documents offline results from last weeks discussions between @CFGrote and @ax3l. @CFGrote can you please edit to correct wrong and add missing information? :) )

Beamline Elements

Currently, calculators act as the algorithmic building blocks describing subsequent stages in the beamline. The following calculators are currently known, all prefixed with photon_:

• source:
  • description: representation of the light source (e.g. as a wavefront, rays, photon distribution) at the very beginning of the photon experiment, i.e. before any optics or photon matter interaction has happened.
  • example: The code FAST generates 3D (x-y-t) wavefronts at the exit of the undulator in an x-ray free electron laser.
• propagator:
  • description: Propagates the light from the light source through a sequence of optical elements (lenses, mirrors, gratings, apertures ,...) to the sample/target interaction point.
  • example: Propagation of wavefronts by means of Fourier wave optics. E.g. implemented in WPG
• interactor:
  • description: Interaction of the photons with the target or sample. Takes into account elementary processes like absorption, emission, scattering of radiation and secondary processes like collisional ionization and recombination. The end product is the electronic state of the sample/target as a function of time during the interaction with the external light source.
  • example: particle-in-cell simulations, radiation-hydrodynamics simulations, molecular dynamics, and many more ...
• diffractor:
  • description: multiplication of form-factors with incoming field & propagation to ideal detector
  • example: Scattering of x-ray pulses from a molecule or a plasma
• detector:
  • description: adds electric response of detector: multi-photon, Compton, boundary effects, ...
  • example:
• analyzer:
  • description: currently only relevant for single-particle images: calculates the 3D electron density in the sample by solving the inverse scattering problem (scattering-signal -> density) through orientation of 2D diffraction patterns and iterative phasing.
  • example: Expand-Maximize-Compress (EMC) for orientation, Difference-Map for phasing.

Code Coupling

• source and propagation can be modeled in the wave-front approximation via SRW with it's Python Binding and high-level interface WPG. A complementary method is raytracing, e.g. using the
  code Shadow, python bindings via xrt or oasys
• A short-pulse laser-plasma experiment would be modeled with PIConGPU and replaces both interactor and diffractor. The X-FEL beam can be discretized into the photon picture from its wavefront description, see scripts in #37.
• An ideal detector result from PIConGPU (2D openPMD "images") can be fed back into the detector calculator.

Related Questions

• #7
• #8
• #31

Sergey wrote on Sept. 13 2016:
I'm getting some IO error and see only one output file:

inputDir: /data/fhgfs/yakubov/simex_bigcase/5keV_3fs_nz35/pmi
outputDir: /data/fhgfs/yakubov/simex_bigcase/5keV_3fs_nz35/diffr/diffr_test
configFile: /dev/null
beamFile: s2e.beam
geomFile: s2e.geom
rotationAxis: xyz
numSlices: 100
sliceInterval: 24
pmiStartID: 1
pmiEndID: 1039
numDP: 200
calculateCompton: 1
uniformRotation: 1
saveSlices: 0
gpu: 0
Error: rank is not zero: 1
Try using hdf5read instead

But if I use the attached script with one of the pmi files you put into /data/netapp/grotec/5keV_3fs_nz35 I see this error (same if I use the script you send me with your data).

when I use it with pmi_out_0000001.h5 from the test suite it works ok.

I used default vaule of 2 MPI processes. If I use 1 process it hangs. More than 2 I did not try yet.

script:

from SimEx.Calculators.SingFELPhotonDiffractor import SingFELPhotonDiffractor
import time

start = time.time()

diffraction_parameters={ 'uniform_rotation': True,
'calculate_Compton' : True,
'slice_interval' : 100,
'number_of_slices' : 2,
'pmi_start_ID' : 1,
'pmi_stop_ID' : 2,
'number_of_diffraction_patterns' : 2,

'number_of_MPI_processes' : 3,

         'beam_parameter_file' : 's2e.beam',
         'beam_geometry_file' : 's2e.geom',
       }

photon_diffractor = SingFELPhotonDiffractor(
parameters=diffraction_parameters,
input_path='./pmi_out_0000001.h5',
output_path='../output/diffr')

photon_diffractor.backengine()

end = time.time()

print "Elapsed", end-start, " s"

We could also add a spack package (actually: a recipe) to deploy source and build on the system.

I did a simulation on simex and keep the product of slice_interval and number_of_slices, and I found the intensity of the diffracted pattern increased with the bigger number_of_slices.

all parameters present in the simS2E config file and needed by radiationDamage executable should be mirrored in the class.

radiationDamageMPI calls prepHDF5.py in the $PWD. Patch the cpp such that it finds py in a default location.

Provide an interface for the Gregori/Fortmann XRTS code v7.

(This issue documents offline results from last weeks discussions between @CFGrote and @ax3l.)

Description

A simulation is staged in various Calculators in SIMEX, see #40, which are currently executed synchronously. It would be very helpful for long-running simulations, e.g. executed on an other machine such as an HPC Queue, to be "submitted" and "collected" asynchronously.

Proposed Workflow

Calculators are "submitted" and can be "queried" on their current state (submitted/waiting, running/waiting, successful/failed result). When working with a (new) PhotonExperimentSimulation object, one can populate it from previous states to work further on the same beamline.

(I imagine an IPython Notebook for those tasks which can be shut down and later its objects can be re-instantiated. Each command should be)

Concrete Python Implementation

To be determined.

Influence on Synchronous Workflows

No influence, can still be done. One could think of convenience methods like wait_all() that do a busy blocking.

Related Issues/Questions

• #7
• #8
• #40

Some singFEL parameters need more/better documentation. In particular numDP, sliceInterval, numSlices.

Currently, singfel sources are downloaded from a db link, this sucks. better would be to download from repo [email protected]:chunhongyoon/singfel.git (@chuckie82 's original) or [email protected]:racnets/singfel.git (my fork).
all patches from singfel's CMakeLists.txt should be applied upstream and removed from our build system.

Based on the XMDYN demo that comes with simS2E, implement a XMDYNDemoCalculator class for PMI in the context of single particle imaging

Reference: XMDYN: Ziaja et al., Photonics 2 (2015) 256

see PlasmaXRTSCalculator for an example

Write a calculator that interfaces the EMC orientational reconstruction algorithm.

It would be great to look into Travis or other CI tools for testing. @ax3l can give advice when he comes to Hamburg.

simex_platform should be repeatedly tested in more than one HPC environment during development to avoid unjustified assumptions on the HPC workflow. HZDR can provide access to its HPC resources.

take existing cmake files + patches

mpich
armadillo
hdf5
boost

• implement checks for required 3rd party libs

I'll try to install simex_platform locally in empty container on ubuntu 14.04.

1. The list of all necessary packages for build simex_platform the following:

apt-get update && apt-get install -y wget python2.7 python-numpy python-scipy python-h5py python-setuptools python-matplotlib build-essential libbz2-dev libgsl0-dev libfftw3-dev liblapack-dev cmake unzip 

(unzip not mentioned in Readme. )

2. I'm fail to build simex_platform, because I have not Intel MKL library installed locally.
   Is it any way to build recon using lapack if MKL absent?

Building recon.
if [ -d  ]; then \
                cd /home/simex_platform/build/../build/externals/src/recon/s2e_recon/DM_Src && \
                make mkl;\
        fi
make[1]: Entering directory `/home/simex_platform/build/externals/src/recon/s2e_recon/DM_Src'
gcc -O3 object_recon.c -o object_recon -m64 -I/include -I/include/fftw -Wl,--no-as-needed -L/lib/intel64 -lmkl_intel_lp64 -lmkl_core -lmkl_sequential -lpthread -lm
object_recon.c: In function 'setup':
object_recon.c:260:8: warning: ignoring return value of 'fscanf', declared with attribute warn_unused_result [-Wunused-result]
  fscanf(fp, "%d %d", &qmax, &num_supp) ;
        ^
object_recon.c:272:19: warning: ignoring return value of 'fscanf', declared with attribute warn_unused_result [-Wunused-result]
             fscanf(fp, "%d", &supp[s][i]) ;
                   ^
object_recon.c:354:11: warning: ignoring return value of 'fscanf', declared with attribute warn_unused_result [-Wunused-result]
     fscanf(fp, "%lf", &intens) ;
           ^
object_recon.c:403:10: warning: ignoring return value of 'fscanf', declared with attribute warn_unused_result [-Wunused-result]
    fscanf(fp, "%lf", &x[i][j][k]) ;
          ^
/usr/bin/ld: cannot find -lmkl_intel_lp64
/usr/bin/ld: cannot find -lmkl_core
/usr/bin/ld: cannot find -lmkl_sequential
collect2: error: ld returned 1 exit status
make[1]: *** [mkl] Error 1
make[1]: Leaving directory `/home/simex_platform/build/externals/src/recon/s2e_recon/DM_Src'
make: *** [/home/simex_platform/build/../build/externals/src/recon/build_dm.stamp] Error 2
root@e68cf649b3ea:/home/simex_platform/build# exit

I ran an example, the result of the orient file is not correct and there is no output file of reconstruction.
There are many NANs in the orient array.

issue.zip

How will the Simex Platform deal with large-scale in-situ scattering simulations that require a sizeable HPC System? Will the workflow pass over values via Python in-situ or via file I/O?

*test workflow with only one file (one source, one pmi traj, one diffraction pattern)
*test workflow with multiple source files, multiple trajectories per source, multiple diffraction patterns per trajectory.

Link:
https://travis-ci.org/ax3l/simex_platform/jobs/131638091#L3626

One of the tests of the day1 beamline segfaults as of commit 6994737

testReadCalculateWrite (SimExTest.Calculators.WavePropagatorTest.WavePropagatorTest)

Test a backengine run with a single input file. ... Optical Element Setup: CRL Focal Length: 31.5955077725 m

23659 Segmentation fault      (core dumped) python Test.py -v

Each special calculator has more or less the same logic for setting up the number and type of output files depending on the type and number of input files. Can this be generalized and moved to either the AbstractPhoton* or even the AbstractBaseCalculator?

Make Propagator class that interfaces wpg.
Input: FAST data
Parameters: wpg beamline
Output: same as XFELPhotonPropagator

Test branch master in travisCI. Needs cherrypicking some commits from develop branch.