Currently we maintain, effectively, 4 distinct dicts:

1. https://github.com/leofang/ptycho_gui/blob/b64326d7d0db5f05bda684a55f295bf1fdeb57e2/nsls2ptycho/core/ptycho_param.py#L23
2. An instance of the above Param() class here: https://github.com/leofang/ptycho_gui/blob/b64326d7d0db5f05bda684a55f295bf1fdeb57e2/nsls2ptycho/ptycho_gui.py#L104-L108
3. configparser: https://github.com/leofang/ptycho/blob/b0b39bd849ad65842659e29dc446f9629fa1433e/utils.py#L8
4. The backend ptycho_trans class https://github.com/leofang/ptycho/blob/b0b39bd849ad65842659e29dc446f9629fa1433e/ptycho_trans_ml.py#L79

This has caused a significant headache every time we want to pass new attributes from the fronted to the backend: We have to update all the four files above plus https://github.com/leofang/ptycho/blob/master/recon_ptycho_gui.py to get it work, which is too tedious from the maintenance point of view.

The goal is to maintain a single dictionary across both frontend and backend. Steps to resolve this are:

• Instantiate the backend ptycho_trans class directly in the frontend, and dump its attributes to disk as before
• Eliminate the intermediate Param class
• Investigate the possibility of making the instance in-memory for the backend to reuse
• Eliminate the usage of configparser

Checklist

• Bump software version
• Write the changelog for the new release
• Assign relevant issues and PRs to the corresponding milestone
• Regenerate the C files from the Cython sources (if they are changed)
• Update the submodule for the backend (if it is changed)
• Mirror both the frontend and the backend to the NSLS-II internal Gitlab
• Update NSLS-II Conda recipe and build the Conda package

mainly for testing on DGX-2, but could also be useful for general users?

...such as NSLS-II/PyXRF#191.

Although the Python mmap doc says

class mmap.mmap(fileno, length, flags=MAP_SHARED, prot=PROT_WRITE|PROT_READ, access=ACCESS_DEFAULT[, offset])
(Unix version) Maps length bytes from the file specified by the file descriptor fileno, and returns a mmap object. If length is 0, the maximum length of the map will be the current size of the file when mmap is called.

When it comes to Mac OS X, the length=0 rule seems to not respected and causes an OSError: [Errno 22] Invalid argument. We should instead set length to be the size of the allocated shared memory, available through shm.size. See, e.g., https://github.com/osvenskan/posix_ipc/blob/46eae821f56f364a3afd6d68b3562adeced3794a/tests/test_memory.py#L135-L165.

A special method should be added such that when clicking the x button on the main window, all windows will be closed (preferably with a confirmation dialogue). This would eliminate the nuisance of multiple clicking every time.

Related: NSLS-II-HXN/profile_collection/pull/15, NSLS-II/playbooks/pull/188

HXN_databroker.py needs to be changed. Also ptycho_gui.py.

This can be done by including the cython-generated C codes in the backend.

Currently the GUI is tied to @NSLS-II-HXN, but other beamlines (such as @NSLS-II-CSX) may also want to use it. We should define some API for flexible coupling to beamline-specific databroker processing functions.

Part of the work will involve creating a switch of some kind to select beamline profiles. Also, defer imports of beamline-specific packages to the corresponding databroker wrappers (e.g., don't import hxntools when the main window starts).

See leofang/ptycho#31.

For better flexibility, the GPU buttons (0, 1, 2, 3) should be replaced by a text field so that arbitrary selection of GPUs (ex: 0, 2-4, 6, 8; similar to how batch processing is specified) can be done. This also frees the limitation of 4 GPUs and avoids the need of auto detection (by giving the users full control).

This is needed until cupy/cupy/pull/1942 is merged to master and released on PyPI.

See leofang/ptycho/issues/23.

Checklist

• Bump software version
• Write the changelog for the new release
• Assign relevant issues and PRs to the corresponding milestone
• Regenerate the C files from the Cython sources (if they are changed)
• Update the submodule for the backend (if it is changed)
• Mirror both the frontend and the backend to the NSLS-II internal Gitlab
• Update NSLS-II Conda recipe and build the Conda package

A number of examples:

• disable algorithms not yet implemented in GPU when the GPU flag is checked
• disable algorithms that do not support multi-mode (or multi-slice) if the corresponding flag is checked
• warn for too large GPU batch size by estimating the GPU memory usage

This is mainly for avoiding supplying a MPI machine profile. Any thought toward this is useful.

need to work with @xiaojinghuang on this

In setup.py (and/or accompanying codes) we should:

• detect existence of GPU and test if CUDA driver & toolkit are properly installed
• detect existence of MPI library
• detect existence of FFTW library
• compile the CUDA code into ~/.ptycho_gui/ (can be avoided if we use RawKernel.compile() in the backend, see cupy/cupy/pull/1889); this would also involve moving configure.sh to the backend.
• detect current beamline and install beamline-specific packages for accessing databroker
• Choose the right CuPy version to install if GPU is to be used
• other things?

@xiaojinghuang requested this feature so that once we identify the ROI and hot pixels for the head of a series of scans, we can use batch mode to extract data from databroker (using the same ROI and hot pixels) and run the reconstruction.

Which is better, robust, and perhaps backward compatible?

1. from @xiaojinghuang's save_data_h5_mll_new.py

    if np.abs(angle) <= 45.:
        x = np.array(df['dssx'])
    else:
        x = np.array(df['dssz'])
    y = np.array(df['dssy'])
    points = np.zeros((2,num_frame))
    points[0,:] = x#[:500]
    points[1,:] = y#[:500]

2. Current approach:
   https://github.com/leofang/ptycho_gui/blob/e1103377937fdfc89692096824c42b3df17b6801/nsls2ptycho/core/HXN_databroker.py#L40
   https://github.com/leofang/ptycho_gui/blob/e1103377937fdfc89692096824c42b3df17b6801/nsls2ptycho/core/HXN_databroker.py#L44
   https://github.com/leofang/ptycho_gui/blob/e1103377937fdfc89692096824c42b3df17b6801/nsls2ptycho/core/HXN_databroker.py#L85-L89

Attn: @xiaojinghuang

Checklist

• Bump software version
• Write the changelog for the new release
• Assign relevant issues and PRs to the corresponding milestone
• Regenerate the C files from the Cython sources (if they are changed)
• Update the submodule for the backend (if it is changed)
• Mirror both the frontend and the backend to the NSLS-II internal Gitlab
• Update NSLS-II Conda recipe and build the Conda package

If the GUI crashes very violently, it may be possible that the shared memory is not cleaned up. Try do a search (based on ownership) and clean up at GUI start time?

Challenges:

• this seems to work only for Linux (with /dev/shm)?

take screenshots, list steps for reconstructions, explain meanings of each button and field, etc

Using shared memory for only two integers is clearly an overkill...

Accompanying issue of leofang/ptycho/issues/12.

In 770b7d2 the Windows menu is disabled: https://github.com/leofang/ptycho_gui/blob/770b7d27860083d7033a3457e971676fc9158026/nsls2ptycho/ptycho_gui.py#L129-L133
This should be worked out so that even if the user accidentally closes those windows, they can still be brought back by clicking the menu items.

Currently when we plot the scan points we assumed the scanned area is a square:

https://github.com/leofang/ptycho_gui/blob/787a35eed043acacd41c25ac56d8b0564c2d5c79/nsls2ptycho/core/widgets/mplcanvas.py#L148

But this may not be always valid...

...because it's currently one of the slowest part in the whole process. This is connected to the save_recon() function in the backend.

On Mac OS X the default maximum size for each shared memory segment is 4MB:

leofang@Leos-MacBook-Pro:~/$ sysctl -A | grep shm
kern.sysv.shmmax: 4194304
kern.sysv.shmmin: 1
kern.sysv.shmmni: 32
kern.sysv.shmseg: 8
kern.sysv.shmall: 1024
machdep.pmap.hashmax: 21
security.mac.posixshm_enforce: 1
security.mac.sysvshm_enforce: 1

leofang@Leos-MacBook-Pro:~/$ ipcs -M
IPC status from <running system> as of Wed May 15 11:09:08 EDT 2019
shminfo:
	shmmax: 4194304	(max shared memory segment size)
	shmmin:       1	(min shared memory segment size)
	shmmni:      32	(max number of shared memory identifiers)
	shmseg:       8	(max shared memory segments per process)
	shmall:    1024	(max amount of shared memory in pages)

This will have to be considered if the reconstructed arrays are too large.

That being said, so far I don't find this causes any problem:

import posix_ipc
s1 = posix_ipc.SharedMemory('/xxx', flags=posix_ipc.O_CREAT, size=4194304+1) # 1 byte larger than maximum

Refs:
http://www.spy-hill.com/help/apple/SharedMemory.html
https://stackoverflow.com/questions/36595754/shared-memory-folder-in-mac-error-shm-invalid-argument

For the Fermat scan (or other non-mesh scans), x/y step size is technically ill-defined. Perhaps change the window elements dynamically based on scan type? It's probably too much work though...

Checklist

• Bump software version
• Write the changelog for the new release
• Assign relevant issues and PRs to the corresponding milestone
• Regenerate the C files from the Cython sources (if they are changed)
• Update the submodule for the backend (if it is changed)
• Mirror both the frontend and the backend to the NSLS-II internal Gitlab
• Update NSLS-II Conda recipe and build the Conda package

Starting from v2.9.0, h5py supports in-memory H5, which means one can allocate a segment of shared memory, within which a H5 object can live. If the frontend can do this, it will help avoid reading large data from disk in the backend.

Thoughts:

• to actually save time, we want to reuse the same H5 many times before closing the shared memory. Is this the most common use case?
• before killing the in-memory object, should we write it to disk? (I think so)

This is a follow-up of #26. The following code snippet was commented out because it would freeze the whole window:

If a more recent Qt/PyQt5 is available (at least 5.10) we should revisit this problem. I suspect it's the underlying implementation causing the problem.

Some were planned with @xiaojinghuang but we didn't have time to implement it...

Checklist

• Bump software version
• Write the changelog for the new release
• Assign relevant issues and PRs to the corresponding milestone
• Regenerate the C files from the Cython sources (if they are changed)
• Update the submodule for the backend (if it is changed)
• Mirror both the frontend and the backend to the NSLS-II internal Gitlab
• Update NSLS-II Conda recipe and build the Conda package

Observation from #20

matplotlib v3.0.3 doesn't seem to behave properly on line plots. The errors aren't plotted at all.

This is due to an unexpected code path introduced in 1d873d4. After initializing the mmap, the string "init_mmap" was mistakenly passed to the error windows, so subsequent data were (silently) rejected.

This does no harm:

Not sure if this is related: every time the start button is clicked, the following warning is always shown:

'c' argument looks like a single numeric RGB or RGBA sequence, which should be avoided as value-mapping will have precedence in case its length matches with 'x' & 'y'. Please use a 2-D array with a single row if you really want to specify the same RGB or RGBA value for all points.

Currently only intensity frames and scan positions are read from H5 in the backend, and the rest is read from the GUI. Therefore, it is possible that we are asking for more info than we actually need for completing the reconstruction.

TODO:

• determine what belongs to "irrelevant information"
• remove those elements from both frontend and backend (related: #7)
• modify the functions involving reading data from databroker and writing to H5 (related: #9)

TODO:

• add at least two more panels so that we can see amplitudes and phases for probe and object simultaneously
• add a panel for showing how scanning points are split across MPI processes (ex: paint them in different colors)
• add a panel for plotting the psi error

Currently the following tasks are done manually but should be automated:

• Mirror the frontend to https://gitlab.nsls2.bnl.gov/leofang/ptycho_gui and backend to https://gitlab.nsls2.bnl.gov/leofang/ptycho
• Update the Cython-generated C file when the .pyx source file is updated
• Tag new releases with changelog input from PR/commits
• other things? tests?

This is an important info that's missing currently. GUI needs to detect this automatically. Perhaps these were not stored in the databroker? Need to check.

Related: #8

@xiaojinghuang reported that the cropped images do not seem right starting this Tuesday or Wednesday. While the production code was not changed this week, it might be related to the recent fix 96d5586.

A number of HXN-specific setups are deeply buried in the GUI. To extend the software they need to be modularized. This is the first step toward enabling different beamline configurations.

Relevant: #10, #32.

TODO:

• alpha is really nothing, don't keep it there
• add parameters used by ePIE
• also remove the fields that are already disabled

This is to work with line_profiler which is currently activated manually in the backend.

As per requested by @mrakitin.

Currently we pass num_processes to ScanWindow:
https://github.com/leofang/ptycho_gui/blob/6e9ea6d5aea1a5f3642fd1acc20a8ba2c2d62718/nsls2ptycho/ptycho_gui.py#L550
However, if the MPI machine file is used, the actual MPI size is counted during initialization:
https://github.com/leofang/ptycho_gui/blob/9e62e62f409b495e83c8a24640f221704990eb58/nsls2ptycho/core/ptycho_recon.py#L99-L125
This workflow needs to be changed/refactored so that the correct size is passed to ScanWindow.

May be relevant: #21