Will Integrant support parallel initialization? about integrant HOT 7 OPEN

In theory that possible, but it would take a bit of work to achieve. Right now the components are sorted and started serially; we'd need a more sophisticated system to figure out which could be started in parallel.

from integrant.

@weavejester how sophisticated would this be? I assume you would need to model the dependency as a directed acyclic graph, then traverse it, starting from nodes with no incoming edges, and at each step loading all selected nodes in parallel? Or am I missing something?

from integrant.

You'd also need to handle threading, Integrant might need to provide a thread pool, or let the user provide one. It seems possible, but would need some thought to keep it easy for the user.

from integrant.

The dependencies are already modelled as a graph, which can be retrieved with the dependency-graph function. However, there are a few difficulties around initiating concurrently:

1. The order of components starting is no longer deterministic
2. We need to build our own iteration mechanism
3. We need to keep a list of all dependencies that have begun to be initiated, as we can run into the same dependency multiple times even if the graph is acyclic
4. We need to halt all threads on an exception, and return the exception
5. We might need some manner of thread pool
6. Certain keys will take a very short time to initiate; these may actually be slower with the thread overhead
7. Maybe other things I haven't thought of

from integrant.

From Erlang Programming Rules:

3.12 Make code as deterministic as possible

For example, suppose one process has to start five parallel processes and then check that they have started correctly, suppose further that the order in which these five are started does not matter.

We could then choose to either start all five in parallel and then check that they have all started correctly but it would be better to start them one at a time and check that each one has started correctly before starting the next one.

Well, probably the default should be start serially.

Only if is really needed, it could explicitly be asked to start something in parallel.

from integrant.

My solution for this problem https://gist.github.com/serioga/1c2f5a6ef771a267a6e726a066e2bf39

from integrant.

Parallel initialization addon for integrant https://github.com/serioga/webapp-clojure-2020/blob/master/src/lib/integrant/async.clj

from integrant.