maxfischer2781 / chainlet Goto Github PK

The behaviour of StopTraversal is not consistent in send, nested or parallel execution. The described purpose is

• Stop the traversal of a path, with either
  • [1] No return value
  • [2] a final return value

Processing multiple sub-chains in chains fails to do [2], treating early return values as regular ones. A bundle behaves similarly.

A bare send will retry indefinitely in case of [1], until either a result is produced (regularly or via [2]) or StopIteration. This easily creates an infinite loop for inappropriate input, or non-defensive chains.

When trying to link to an unwrapped generator of function, these can be automatically wrapped.

Wrappers created from decorators, such as funclet or genlet, are missing functionality:

• Cannot be pickled. The wrappers hide their slaves in the global namespace. This means that wrappers must define how slaves are pickled.

• Cannot be set to fork/join. The respective flags are set only on the factory, but are not properly taken over to the resulting chainlet.

• Wrappers cannot be introspected properly. They show up only as a generic function.

Chain traversal currently uses an arbitrary but strictly synchronous traversal method that is piecewise depth-first. For asynchronous, lazy and parallel traversal, an asynchronous model is required. Ideally, this does preserver the result of synchronous traversal.

The abstraction to chains and graphs does not expose concurrency: only a top-level fork of producers can be detected and requires inspection of the chain.

A possible solution is for chains/graphs to provide paths - a collection of each sequence the chain/graph encodes. For example, the graph a >> (b, c) creates the paths [ a >> b, a >> c]. A path is a linear chain, and a linear chain provides itself as its only path.

Paths could be the entities that drivers work on, allowing the appropriate parallelisation. This should also allows asyncio drivers.

With separate Chains and Links, there is currently no way for a Link to signal an end of chain traversal. This makes filters impossible, and thus makes future loops questionable.

There are two situations for aborting the traversal of a chain:

• A producer is exhausted, and the chain can never be traversed again. For symmetry with chains as generators, the StopIteration is appropriate.
  • This may require chains to remove subchains that are exhausted, as long as there are other, parallel chains that are not exhausted. A chain would re-raise StopIteration if there are no more subchains.
• An element decides to block further traversal based on its current data chunk, but may be traversed again with other input. A new exception, StopTraversal, is suggested.
  • This exception should not "bubble up" to iterations. A separate interface for traversal and iteration may be needed.
    • The public interface should remain as send and __next__, since this is used by external iteration. It uses traversal, but filters out any internal exceptions.
    • An internal interface for traversal should be added, e.g. as visit(value=None), or a conventional chain_* (chain_send), link_*, c*.

The current interface has some shortcomings:

• pull mode cannot be done externally, making TRE impossible. Since __next__ cannot take any parameters, it must be used inside the chainlet.
• there is an either-or to pull-push, which the syntax does not imply.
• implementing both pull-push duplicates code.
• the simile with generators is not preserved, where __next__() is equivalent to send(None).

Suggested new interface:

• move all handling of values to send(value), which is consistent with the generator interface.
• calling __next__() is equivalent to send(None).
• implement explicit push(value) and pull(value) which use send to process their input.

Dispatching to specific chains for specific values allows for an easy switch/dispatch/fork/... behaviour. The dict class and its literals could easily represent it:

pre_chain >> {
  # key => chain dispatches
  str: int,
  float: round,
  # how to transform a value to derive a key
  '__transform__': type,
  # what to do if the key matches no case
  '__default__': lambda value: value  # do nothing if no key matches
} >> post_chain

• Map value => chain, then push value along chain
• Settings via __double_underscore__ names

There should be an accessible way to define how chainlets of the same or different types can be merged.
Given a chainlet like

@funclet
def multiply(value, by=4):
    return value * by

it is desirable to ensure

multiply(a) >> multiply(b) == multiply(a * b)

is automatically ensured. There is currently no simple way to do this.

It is currently possible to overwrite ChainLink._link, to act before the Linker is run. Obviously, this requires overwriting a hidden method.
A more plugable mechanism that also works cleanly for funclet and co would be preferable.

• Use a special method, such as __compile__, __merge__ or __link__
• Ensure it is run consistently with the linker
  • Should be skipped if __merge__ is None (same as for __hash__)
  • This could replace/supersede a linker run
• Allow setting the method on funclet
  • Probably similar to property.set, e.g. multiply.merge

Chaining chainlets directly is not suitable for forking chains. Currently, chains inside a fork are not created as expected:

start >> a >> (b1 >> b2, c1)

Creates the links:

start >> a
a >> c1
b1 >> b2
a >> b2

In this case, b1 is never used. The last link should have been a >> b1. It may be suitable to have a (temporary?) overarching chain object.