ke0m / scaas Goto Github PK

Currently I just print to stdout. When I change the way of launching workers to multinode, my log viewer will not work the same way. I need to add a logger so that each process might write its log to a file.

I currently have a 3D Perlin noise function that allows for the creation of velocity anomalies. While it works great in both 2D and 3D, it is quite slow in 3D. Porting to C++ and parallelizing with OpenMP threads will allow for faster anomaly creation

Fix directory structure so that follows a standard python directory structure (setup.py, modules with init.py, etc)

Currently workers are called with no arguments (they also do not have a main). I would like to add a main and also arguments (for example the server to which they should connect) and potentially other arguments.

I have a new implementation of the complex square root that gives 3X speed up in ssr3ssf kernel. Needs to be merged into master

Reformat the code so private variables have only one underscore. Also use yapf for reformatting

Would be nice to put together a notebook that shows the processing and imaging of the BEI dataset (from nearly raw data to image).

Based on my limited research, it seems that changing from pyzmq to mpi4py should not be too difficult. Also, mpi4py has many advantages including:

1. Functions that block up arrays and allow for large copies (scatterv, gatherv)
2. Collective operations such as reduce and all reduce
3. Very simple API whereas with ZMQ I have done most everything by hand

In addition, mpi4py does not require pickling of python functions (as did DASK) so there should not be any problems there with pybind11

Edit: Something to consider here is how MPI handles node pre-emption/failure vs ZMQ. With ZMQ there different patterns that can be employed that allow for handling this smoothly. I am not certain if this can be down with MPI

The IO can also be split into a small repo.

Need to do new full benchmark to see performance gain

The linear and non-linear optimization code can be split into its own repo

When migrating the F3 dataset, I do full inline and crossline aperture. While I think it is necessary to do full inline aperture, the crossline aperture can definitely be windowed and change as the shot moves in the crossline direction

Currently, I launch remote workers with a python script. I would like to change it follow what is done in other libraries:

python -m module.worker