pveber / bistro Goto Github PK

It seems mounts as they are done in engine/Task do not work on MacOS X, probably because Docker runs in a virtual machine, which doesn't host the files

• docker commands should not be executed outside docker (suppress use_docker option in Bistro_app)
• commands should specify a shell
• new task that would just create a file with a given contents specified as an expression (for config files or scripts) --> done with e722f2e

paper

It would be nice to have a mechanism to remove intermediary targets during the execution of the workflow, to save space in the cache.

Upstreaming this from Nixpkgs:

# File "ppx/bistro_script.ml", line 201, characters 18-41:
# 201 |           let e = Parser.parse_expression Lexer.token (Lexing.from_string txt) in
#                         ^^^^^^^^^^^^^^^^^^^^^^^
# Alert deprecated: module Ppxlib.Parser
# Accessing this module directly is deprecated, use Ocaml_common.Parser instead
# File "ppx/bistro_script.ml", line 201, characters 18-41:
# 201 |           let e = Parser.parse_expression Lexer.token (Lexing.from_string txt) in
#                         ^^^^^^^^^^^^^^^^^^^^^^^
# Error: Unbound value Parser.parse_expression

Indeed Nixpkgs simply patches Parser.parse_expression to Ocaml_common.Parser.parse_expression here.

Bistro functions may have arguments that are not workflows. In that case, the workflow's id should depend also on the value of these arguments, which cannot be expressed at the moment. Let's call these arguments parameters.

... that shows they've been run from bistro.

for instance getting inspiration from https://pilsniak.com/how-to-install-docker-on-mac-os-using-brew/

Currently all commands executed through docker are executed as root (in the container), which has (at least) two drawbacks:

• it creates output files owned by root on the "host" file system
• some tools (like meme in mpi mode) refuse to be run as root.

One solution would be to execute command through docker after changing user in the container, choosing the same uid than the (bistro) process owner.

When several tasks require significant bandwidth, it would be better to run them one after the other, to avoid too much idling. One solution would be to have a flag to say that a workflow consumes mostly network bandwidth and add network bandwidth as a limited resource.

Suggestion for a phylogenetic pipeline by @Boussau:

Items of the same level are alternatives

clustering:

• blast then
  • orthomcl
  • silix (+hifix)

alignement:

• mafft
• clustal-omega
• muscle
• prank
• Saté
• Pasta

gene phylogeny:

• PhyML
• RaxML
• IQTree
• GARLI
• mrBayes
• Paup
• Beast
• fasttree
• revBayes

species phylogeny:

• MPEst
• Astral

Singularity offers ro or rw mounts, with the syntax:

singularity shell --mount /host:/guest:ro ...

Would be nice to use for cache and inputs.

https://coreos.com/rkt/
https://coreos.com/rkt/docs/latest/rkt-vs-other-projects.html

Expected benefits:

• don't run as root
• don't log stdout and stderr

Looking at content of files, looks like software is GPL.

Would be nice to get in README, or in a LICENSE file, the license of the software to inform users without needing to dig in files
:-)

Alternative to htseq-count

Publication

• ocaml primer
  • named and optional arguments
• bistro
  • selectors
  • inputs

The repository generated by Bistro_app.with_backend is populated with absolute links to the cache, but this prevents a move of the containing directory. It would be better to switch to relative links.

Would be convenient to have a mechanism general allowing to inject run-time values in workflow scripts. Scripts can then be slightly customized while not changing their ID. Variables like np and mem could be reimplemented this way, and that would be a nice way to introduce the name of the currently run executable (to implement custom procedures in a portable way).

For large workflows with many shared sub-workflows, the serialization via s-expressions produces a very large representation (exponential?). This hurts in particular when saving a workflow description in the database (currently only in the statistics table).