njwilson23 / depgraph Goto Github PK

DependencyGraph methods should be able to detect cases in which dependencies are circular, as in

graph.add_relation(a, (b,))
graph.add_relation(b, (c,))
graph.add_relation(c, (a,))

or even

graph.add_relation(a, (a, b, ...)

In this case, probably makes sense to raise something like a CircularDependencyError exception.

            ...
            i, dep = queue.pop(0)
            marks[dep] = i
            ...

Otherwise I'm not seeing any code that would produce items that don't have any parents?
Simple code:

from depgraph import Dataset, buildall
parent = Dataset('parent-does-not-exists', tool=None)
child = Dataset('child-does-not-exists', tool=None)
child.dependson(parent)
for stage in buildall(child):
    for dep, reason in stage:
        print("Building {0} with {1} because {2}".format(dep.name, dep.tool, reason))

Output only builds child, not the parent:

Building child-does-not-exists with None because the dataset doesn't exist

This is critical. Parallel builds should permit datasets that require a build and don't depend on each other to be constructed at once. This might work by having a DependencyGraph.parallel_needsbuild method that returns lists of Dependencies that can be built. Alternatively, could try something concurrent where a channel is filled with eligible builds as the dependency graph is traversed.

It is probably best to leave resource allocation to the function managing the build, but each dependency could have an attribute specifying the number of threads that can be used to build it.