Currently, most of the kernels can run a single rank version (often useful for debugging) except for UnboundedPoissonSolverMPI2D. This is caused by an assumption within that class that fails to cover the case when the grid is not distributed at all (i.e. grid_topology = [1, 1]).

We need a field boundary penalisation kernel for 2D fields. I believe we do not need this for periodic boundary cases, so the implementation should then check for periodicity conditions in mpi construct before proceeding. One thing that I am not sure about in the implementation in sopht-backend kernel generator returns a kernel that operates on all sides, whereas here we need to selectively operate on each side depending on the coordinate of the block of field we are operating on. Any thoughts on how we retrofit sopht-backend implementation here would be appreciated @bhosale2 thanks.

Currently, mpi supported plot utility in sopht-mpi supports only contour plots. We need line and/or scatter plots to facilitate plotting immersed body in quick visualizations.

During the refactoring of examples to use the new position field convention, we missed out on this example case. I will patch this example and push a PR briefly.

I am thinking of internalizing the get max of vorticity field functionality into flow simulator itself. That way one can track diagnostics of max vorticity and the global max is taken care of within the flow simulator itself (with reduction and whatnot between ranks). What do you think @bhosale2 ? A similar idea here is the internalization of get lag grid deviation error in immersed body flow interactor.

We need to have IO tools for dumping fields for visualization and post-processing. I am planning to build on top of distArray functionality. It's something we need later on, so putting this as medium priority for now.

Toward the development of a 3D flow simulator, we need 3D mpi version for the following kernels, utilizing the backend kernels in sopht-simulator.

Poisson solve

• #131
• #133

Stencil operation

• #103
• #104
• #108
• #109
• #113
• #95
• #99
• #112
• #130
• #123
• #124
• #128
• #129

It seems that sopht-backend had an update on typing for pystencils. We need to patch our code here accordingly.

Currently, MPIFieldCommunicator() assumes rank=0 holds all the global field. We need something that allows for flexibility in setting a custom master rank. This comes as a feature useful for MPIPlotter2D() later since the lagrangian grid is allowed to be owned by a custom master rank (i.e. not rank 0).

We need a cleaner solution than print statements in logging outputs.

An idea would be to create a MPI-logger that works the same as logging in Python from a user perspective. The MPI-logger gets initialized and depending on what mode is used, records streams from each rank or from a single rank (typically the root rank = 0) and to either sys output or a file. An example use style I envision is something like below. If the idea makes sense, I will put together the implementation soon. Please let me know what you think @bhosale2 . One thing I am not sure about the structure I'm suggesting is where do we set the logger config (levels, filename for output streams, etc). Please also let me know if you have a better suggestion. Thanks.

# mpi_logger.py
class MPILogger:
    def __init__(self, ...):
        self.logger = logging.getLogger()
        if rank == 0:
            self.verbose = True
        else:
            self.verbose = False

    def debug(self, ...):
        if self.verbose:
            self.logger.debug(msg)

    def info(self, ...):
         ...

    ...

logger = MPILogger(...)

# utils/__init__.py
from .mpi_logger import logger

# some_kernel.py
from sopht_mpi.utils import logger

def some_kernel():
    logger.info("info from some kernel")

# main.py
from sopht_mpi.utils import logger
from sopht_mpi.some_path import some_kernel

some_kernel() # this should print from some kernel
logger.info("info from main")

Currently the MPI utils support only 2d. We need a 3d counter part.

• #3
• #4
• #6

We need the following to perform flow past rod simulation.

• #74
• #75

We need to have a MPI-supported 2D flow simulator, with a similar style employed in SophT-Examples. Once this is done, we can do simple benchmarking for solver accuracy. @bhosale2 I'm thinking we benchmark it for two different cases (one with pure flow, and another with a rigid body). Please let me know if the list below works, and we can update the list as we go on.

Update: I am adding a flow past rod case, just to be able to test moving lagrangian grid as well as to start the process of incorporating pyelastica simulator (flow past cylinder does not need to pyelastica simulator time step, so technically only uses rigid body).

• #45

Benchmark cases

• #52
• #55
• #69

Now that we have a working version of 2d flow simulator with a few example cases, it's probably a good time to refactor the code before we move on to 3d flow simulator. In the next few days, I will include the following changes:

• #82
• #84

Please let me know if there is anything else you think would need refactoring @bhosale2

Toward the development of a 2D flow simulator, we need 2D mpi version for the following kernels, utilizing the backend kernels in sopht-simulator.

Poisson solve

• FFT kernel
• #36

Stencil operation

• #16
• #18
• Diffusion flux
• #14
• #20
• #21
• #30
• #32
• #22
• #34

Currently, we have separate files with separate classes for 2D and 3D MPI utils features. This can be generalized into dimension-agnostic code, so that users will not have to worry about the dimension of the MPIConstruct they are initializing, rather it will be inferred from the grid_size prescribed by the user.

I'm still not fully sure if we will benefit from this, and will see as we go when developing the 3D simulator. I'm putting this as low priority for now.

Things that need refactoring:

• #93
• Dimension-agnostic MPI utils

Currently, some of the test cases for kernels with kernel_support = 0 are indexed with inner_index = slice((kernel_support, -kernel_support)). This would not be doing what we intend for it to do since we basically want to compare the entire field, but end up comparing nothing (with slice((0,0)) we end up with array of size [0, 0]).

For this case, we need forcing grids and would require a few components as below.

• #56
• #58
• #57
• #59
• #72
• #66

Currently in all the test files, while the class name has been renamed to MPIFieldCommunicator, the variable used when creating an instance of the field comm. object is still named as mpi_field_io_communicator. I'll rename all of these to be consistent.

Since sopht-backend will not be maintained and the backend features are now ported to sopht-simulator (see SophT-Team/SophT#72), we should update our dependency here to use backend code from sopht-simulator instead.

@bhosale2 there is currently a bug in the outplane field curl, which went undetected as a consequent of a minor bug in indexing the inner cells of a vector field in the test case. The correctly indexed inner cell will give errors as shown in the figure below, which fails the test.

The good thing here is I know this error comes from the fact that in sopht_backend, the outplane field curl kernel sets the boundary to 0, and this requires some tinkering when dealing with boundaries between ranks (which I know how to deal with, similarly to penalise field boundary kernel).

I noticed that in the sopht_backend kernel, you have marked a TODO for one sided correction or so. Let me know if I understand this correctly, but is there plans to allow for optional correction for different sides? If this is in your plans, then I will still use this kernel in sopht-mpi. Otherwise, I will just code up my own in the current outplane field kernel. Thanks!

We recently found out that new macos was not working due to an internal issues conflicting between numba and pystencils. As a temporary fix for now I think we should remove macos in the CI until this issue is resolved on their end (either macos or numba/pystencils).

Note: For macos user who are facing the same issue due to pystencil/numba conflicts in macos (an example of error message below), this can be temporarily fixed on your end by reinstalling numba using conda install numba

E   ImportError: dlopen(/Users/runner/Library/Caches/pystencils/objectcache/mod_d84531c67f646ef86248d2be72c311f3b177830303be7d2624642eebaf44ff2e.so, 0x0002): symbol not found in flat namespace (___kmpc_barrier)

<frozen importlib._bootstrap>:241: ImportError

We need quick visualization tool for 2D flow field that adopts the matplotlib style. This will be helpful for debugging etc. I will implement a minimal version of this, meant exactly for quick debugging without needing to load files in paraview and so on.

We should have a way to backtrack from the generated kernel the kernel_support defined for that kernel.