becksteinlab / mdworks Goto Github PK

At the moment, we can specify a postprocessing_wf as a keyword argument to make_md_workflow, and this will be appended as a child workflow to the FilePull task. The idea for this is that postprocessing of the resulting trajectory can begin immediately after it's landed at the archive. However, we currently do not have a similar mechanism for appending a workflow to the end of the last *Continue task (one that chooses not to continue).

We should rename postprocessing_wf to postrun_wf, and add a new keyword argument for post_wf. This will give two hooks for postprocessing at different levels, both on a per-run basis and after the simulation has completed.

At the moment, we pull the files resulting from the MD run after it completes, but then we don't actually delete the files on the remote scratch space. For some systems this is fine, but it is better that we clean up after ourselves.

We should add a new Cleanup Firework to the workflow that only executes after a completed Pull.

new address https://materialsproject.github.io/fireworks/

To streamline development and avoid retreads of old bugs (and introducing new ones), we need to start building out a testing framework. We can use the way tests are done in Fireworks for inspiration.

Can use a MongoDB instance on Travis as launchpad, and can test all workflow generating functions/FireTasks defined in this package individually.

A weakness of the current workflows is that if a cluster goes down for some time and later comes back up, staged files on this cluster will likely be stale. One way around this is to add a checksum for all files staged to the StagingTask, and make these checksums available to the next FireWork (which performs MD on the remote cluster). We'll add a task that checks the stored checksums against the files it sees when it executes. If they don't match, we fizzle the firework. This is better than the current situation where execution will proceed forward with stale inputs.

It would be great to get a first pass at a function that produces a diffusion-map directed MD workflow. For a start, this can use gromacs-specific FireTasks, but we hope to eventually make it work with a variety of MD engines. No need to get too ambitious now, though.

The building blocks made along the way will be useful for many other multi-simulation/analysis loop-style methods, so this would be trailblazing as far as this package is concerned.

At the cost of making the workflow a bit more complex, it would be a performance boost to follow up the MD Firework with a MinimalPull FireTask that pulls back only the files necessary to begin the next run (which, unless I'm mistaken, should be the files specified in the files kwarg). These files would need to be sufficient for the ContinueTask of the MD engine being used to make its decision.

Basically, MinimalPull has two children: Continue and Pull. The advantage is that the next job, if necessary, can be ready to go while the data from the last one is still being pulled back to the archive. We don't have to wait for a full transfer to keep going.

We could make workflow configuration optional, since it introduces more complexity and may not be suitable for all MD engines. Something like minimal_pull=True could be set to use it, defaulting to False.

One barrier to using mdworks (and fireworks) is the setup required to make workflows execute on different hosts. Building a directory of templates for the files needed, which can then be simply copied into place on a given resource and configured for that resource by the user with the guidance of inline docs, would be of great help.

We can call this directory of templates the firecrate. Or maybe the burnbarrel.