Replace use of luigi with ketrew about eggo HOT 4 CLOSED

I was gonna ask about this when I see you. The main things I want are code reusability and a make-like ability to figure out what's complete. Luigi has been less good than expected in the reusability.

from eggo.

I was mostly trolling (hence the gif). OCaml's a pretty big hill to climb and Ketrew is still stabilizing. That said we're using it to run pipelines these days and a few of our lab members have gotten familiar enough with it to write their own extensions.

from eggo.

@hammer Thanks for mentioning Ketrew. I agree it would be good to choose a single pipeline execution framework. I think Luigi is a reasonable choice at this point, but we could replace it in the future if the replacement has the properties Uri mentions (plus ability to easily run on a Hadoop cluster).

Luigi is an implementation detail for eggo, so it would be possible to change it without users noticing, since the important thing is having a well-defined way to turn input sources into output datasets.

from eggo.

Closing cuz..... :)

from eggo.