coobas / wowp Goto Github PK

Originally reported by: Jan Pipek (Bitbucket: janpipek, GitHub: janpipek)

A type-annotated function, using the syntax recommended in Python 3.5(?), like this:

#!python

def greeting(name: str) -> str:
    return 'Hello ' + name

FuncActor(greeting)

Cannot be converted to a FuncActor:

#!

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-6-2ae49fe35a1c> in <module>()
      2     return 'Hello ' + name
      3 
----> 4 FuncActor(greeting)

~/code/my/wowp/wowp/actors/base.py in __init__(self, func, args, kwargs, outports, inports, name)
     66         elif isinstance(outports, six.string_types):
     67             outports = (outports, )
---> 68         for name in outports:
     69             self.outports.append(name)
     70 

TypeError: 'type' object is not iterable

We should adapt the FuncActor to deal with this syntax or at least fail with a more meaningful message.

• Bitbucket: https://bitbucket.org/urbanj/wowp/issue/10

Originally reported by: Jan Pipek (Bitbucket: janpipek, GitHub: janpipek)

I think it should be possible to implement. Currently, I don't need it that much but if this feature were possible, it would some of my chagall connections more fbp-ic.

• Bitbucket: https://bitbucket.org/urbanj/wowp/issue/4

Originally reported by: Jan Pipek (Bitbucket: janpipek, GitHub: janpipek)

It is not even possible to "pip download" the package because setup.py depends on its requirements. Catch XXII-like situation :-)

Collecting wowp
  Using cached wowp-0.1.1.zip
  Saved ./wowp-0.1.1.zip
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-vbppnz_8/wowp/setup.py", line 3, in <module>
        from wowp import __version__
      File "/tmp/pip-build-vbppnz_8/wowp/wowp/__init__.py", line 14, in <module>
        from .components import Component, Actor, Workflow
      File "/tmp/pip-build-vbppnz_8/wowp/wowp/components.py", line 10, in <module>
        import future
    ImportError: No module named 'future'
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-vbppnz_8/wowp

• Bitbucket: https://bitbucket.org/urbanj/wowp/issue/6

Originally reported by: Jan Pipek (Bitbucket: janpipek, GitHub: janpipek)

It breaks (and similarly the randomizing one) one of the key concepts of FBP.
What enters, should leave in the same order (unless the component itself explicitly
re-arranges the inputs).

The actors should not be re-entrant (at least not by default) and their queues should be managed by the scheduler properly.

See test in janpipek/wowp repo.

• Bitbucket: https://bitbucket.org/urbanj/wowp/issue/2

Originally reported by: Jan Pipek (Bitbucket: janpipek, GitHub: janpipek)

http://nerds.airbnb.com/airflow/

• Bitbucket: https://bitbucket.org/urbanj/wowp/issue/1

Originally reported by: Jan Pipek (Bitbucket: janpipek, GitHub: janpipek)

Take inspiration in https://github.com/numpy/numpy/blob/master/doc/neps/dropping-python2.7-proposal.rst

• Bitbucket: https://bitbucket.org/urbanj/wowp/issue/9

Something like:

len_actor.inports.append("in_str", type=str)
len_actor.inports.append("out_int", type=int)

Originally reported by: Jan Pipek (Bitbucket: janpipek, GitHub: janpipek)

When you have a "collector" component that does some action only after all input is pushed to its port(s), there should be some way to let it know it's time to make the action.

This could be:

• a FinalizeToken (silently passing through actors that don't use it)
• some kind of port.close() operation,
• some kind of actor.close()
• marking of components as finished by the workflow
• any combination of the previous

• Bitbucket: https://bitbucket.org/urbanj/wowp/issue/3

Originally reported by: Jakub Urban (Bitbucket: urbanj, GitHub: urbanj)

S........................FSSSSSSSS....SS....
======================================================================
FAIL: Test whether more threads are being used by the scheduler
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Miniconda3\envs\wowp-py27\lib\site-packages\nose\case.py", line 197, in runTest
    self.test(*self.arg)
  File "D:\Workspace\wowp\wowp\tests\test_schedulers.py", line 103, in test_ThrededScheduler_creates_threads_and_executes_all
    assert jobs_executed['count'] == branch_count * branch_length
AssertionError

(Python 2.7 on Windows 8.1)

• Bitbucket: https://bitbucket.org/urbanj/wowp/issue/5

Originally reported by: Jakub Urban (Bitbucket: urbanj, GitHub: urbanj)

Orso suggested to add actor connection using the + operator (in analogy to +=). This operation would return the resulting workflow. E.g.,

#!python
workflow = actor2.inports['inp'] + actor1.outports['out']

This is feasible. However, there is currently a problem with workflow separation. If actor2 is connected to another actor, the output of workflow will be empty.

To solve this issue, workflows have to be treated separately (in separate objects) from actors. This should be feasible and might have other advantages compared to the current approach, in which connections are bound to actors.

• Bitbucket: https://bitbucket.org/urbanj/wowp/issue/7

Originally reported by: Jan Pipek (Bitbucket: janpipek, GitHub: janpipek)

I don't like the way Chain (in a naive way) and MapReduce (in a more sophisticated way) create actors and use schedulers. My opinion is that no actor should ever know what a scheduler is.

We should think about another implementation that would allow dynamic creation of actors. I suggest that we create something like "temporary actors" which will be returned by the main actor (@decorated or inheriting DynamicActor) and reused in the workflow scheduler and later deleted.

• Bitbucket: https://bitbucket.org/urbanj/wowp/issue/11

Originally reported by: Jakub Urban (Bitbucket: urbanj, GitHub: urbanj)

Especially for parallel schedulers. The IPyClusterSchedulers basically floods ipyparallel with job.ready() calls. I see two options:

1. Use callbacks
2. Use an event driven library, e.g. Twisted ot Tornado

• Bitbucket: https://bitbucket.org/urbanj/wowp/issue/8

Matching of input and output ports inside the chain can be implemented easily. On the other hand, it seems necessary to have the first input and last output ports explicitly named.

Related to #4

Originally reported by: Jan Pipek (Bitbucket: janpipek, GitHub: janpipek)

It would be good for visibility of the project and involvement of other developers - the Git base is much wider than mercurial one.

• Bitbucket: https://bitbucket.org/urbanj/wowp/issue/12