maxfischer2781 / chainlet Goto Github PK
View Code? Open in Web Editor NEWPython module for linking generators/iterators to processing chains
License: MIT License
Python module for linking generators/iterators to processing chains
License: MIT License
The behaviour of StopTraversal
is not consistent in send
, nested or parallel execution. The described purpose is
Processing multiple sub-chains in chains fails to do [2], treating early return values as regular ones. A bundle behaves similarly.
A bare send
will retry indefinitely in case of [1], until either a result is produced (regularly or via [2]) or StopIteration
. This easily creates an infinite loop for inappropriate input, or non-defensive chains.
When trying to link to an unwrapped generator of function, these can be automatically wrapped.
Wrappers created from decorators, such as funclet or genlet, are missing functionality:
Cannot be pickled. The wrappers hide their slaves in the global namespace. This means that wrappers must define how slaves are pickled.
Cannot be set to fork/join. The respective flags are set only on the factory, but are not properly taken over to the resulting chainlet.
Wrappers cannot be introspected properly. They show up only as a generic function.
Chain traversal currently uses an arbitrary but strictly synchronous traversal method that is piecewise depth-first. For asynchronous, lazy and parallel traversal, an asynchronous model is required. Ideally, this does preserver the result of synchronous traversal.
The abstraction to chains and graphs does not expose concurrency: only a top-level fork of producers can be detected and requires inspection of the chain.
A possible solution is for chains/graphs to provide paths - a collection of each sequence the chain/graph encodes. For example, the graph a >> (b, c)
creates the paths [ a >> b, a >> c]
. A path is a linear chain, and a linear chain provides itself as its only path.
Paths could be the entities that drivers work on, allowing the appropriate parallelisation. This should also allows asyncio drivers.
With separate Chains and Links, there is currently no way for a Link to signal an end of chain traversal. This makes filters impossible, and thus makes future loops questionable.
There are two situations for aborting the traversal of a chain:
StopIteration
is appropriate.
StopIteration
if there are no more subchains.StopTraversal
, is suggested.
send
and __next__
, since this is used by external iteration. It uses traversal, but filters out any internal exceptions.visit(value=None)
, or a conventional chain_*
(chain_send
), link_*
, c*
.The current interface has some shortcomings:
__next__
cannot take any parameters, it must be used inside the chainlet.__next__()
is equivalent to send(None)
.Suggested new interface:
send(value)
, which is consistent with the generator interface.__next__()
is equivalent to send(None)
.push(value)
and pull(value)
which use send
to process their input.Dispatching to specific chains for specific values allows for an easy switch/dispatch/fork/... behaviour. The dict
class and its literals could easily represent it:
pre_chain >> {
# key => chain dispatches
str: int,
float: round,
# how to transform a value to derive a key
'__transform__': type,
# what to do if the key matches no case
'__default__': lambda value: value # do nothing if no key matches
} >> post_chain
value
=> chain
, then push value
along chain
__double_underscore__
namesThere should be an accessible way to define how chainlets of the same or different types can be merged.
Given a chainlet like
@funclet
def multiply(value, by=4):
return value * by
it is desirable to ensure
multiply(a) >> multiply(b) == multiply(a * b)
is automatically ensured. There is currently no simple way to do this.
It is currently possible to overwrite ChainLink._link
, to act before the Linker
is run. Obviously, this requires overwriting a hidden method.
A more plugable mechanism that also works cleanly for funclet
and co would be preferable.
__compile__
, __merge__
or __link__
__merge__ is None
(same as for __hash__
)funclet
multiply.merge
Chaining chainlets directly is not suitable for forking chains. Currently, chains inside a fork are not created as expected:
start >> a >> (b1 >> b2, c1)
Creates the links:
start >> a
a >> c1
b1 >> b2
a >> b2
In this case, b1
is never used. The last link should have been a >> b1
. It may be suitable to have a (temporary?) overarching chain object.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.