opensistemas-hub / osbrain Goto Github PK

Currently, timers are blocking on first call...

Currently:

ns = NSProxy(...)
agent = ns.proxy('agent_alias')
address = agent.addr('socket_alias')

Proposed::

ns = NSProxy(...)
address = ns.addr('agent_alias', 'socket_alias')

On system interruption, stderr should not be flooded with useless output.

Agent code is starting to be a little mess right now. We should refactor sockets to inherit from zmq.sockets (we probably need to subclass zmq.Context too, then).

Sockets should store address information to avoid looking up tables (dictionaries) and to reduce the agent's number of attributes.

The API should be more homogeneous. In example, instead of multiple if-else cases, implement .send() methods in the sub-classed sockets that take care of peculiarities or special cases.

A changelog file should be created starting on v1.0.0 to track future changes.

• About
  • Feature overview
  • What can you use osBrain for?
  • Performance
• Introduction
  • Installation
  • Hello world
  • Agents and Proxies
  • The name server
• Basic communication patterns
  • Push pull pattern
    • Example
    • List of handlers
  • Request reply pattern
    • Example
    • Yield vs. return (reply early)
  • Publish subscribe pattern
    • Example
    • Filtering
• Other considerations
  • Clients versus servers
  • Adding new methods
  • Lambdas
  • Shutting down
  • OOP
• Other patterns? (i.e.: stream)
  • ...
• Timers
  • Repeated actions
  • Delayed actions
  • Closing timers (#38, #22)
• Transport protocol (#37, #25)
  • Introduction
  • Global configuration
  • Per-agent configuration
  • Per socket configuration
• Serialization
  • Introduction
  • Global configuration
  • Per-agent configuration
  • Per socket configuration
• Advanced proxy handling
  • Understanding proxies better
  • Executing unsafe calls
• Distributed systems (#4)
  • Accessing agents from another terminal through proxies
  • Example of distributed systems in the documentation
• Security
• Developers
  • Workflow
  • Installing dependencies
  • Running the tests
  • Generating documentation
• API
• Software license and disclaimer

By default proxies should use safe remote calls (i.e.: through inproc). This, though, should be configurable.

• Create an UnsafeProxy class?
• Use proxy.unsafe.<methodcall>?
• Use unsafe=True parameter?

Steps:

1. Make all current tests pass changing remote calls to safe (no new functionality).
2. Create new tests to verify and implement the above functionality.

Add documentation on potentially blocking calls and ways to avoid them:

• Changing the multiplexed server to a threaded server.
• Using other communication patterns (i.e. PUB-SUB).
• Using timers or changing the main loop.
• Using one-way calls (explain the risks: one-way -> concurrency -> inproc is better -> timers).
• Using a separate script/terminal for the configuration.

osBrain should provide a convenient synchronizartion method when using PUB-SUB patterns. This is very useful for differential updates (i.e.: time series in osMarkets).

• Accessing agents from another terminal through proxies
• Add an example of distributed systems in the documentation.

Have a look at:

• https://codecov.io/
• https://codeclimate.com/
• https://coveralls.io/
• https://scan.coverity.com/

Features, advantages, disadvantages...?

Implement two step interruption for agents/name server. First SIGINT captured results in clean exit/kill (i.e.: agent waits for code execution and tear-down and name server waits for all agents to be unregistererd). If another SIGINT is captured, kill with no mercy (as we are doing it right now).

Some examples which do not appear in the online documentation may be outdated. Review and update them:

• dipatcher
• ventilator

Implement timers for agents.

Maybe merge set_method() and set_attr() into set()?

Or rather make the proxy able to set attributes and methods automatically with proxy.x = y?

Create tests to verify behavior on interruption signals. In example, an interrupted system should end up with all agents and name server killed (no Python process should linger).

See 7cb8c6f (revert) for an example of how agents can remain alive after SIGINT.

#25

Documentation on serialization should include instructions on how to change the serialization method (i.e.: global setting, per-agent configuration and per-socket configuration).

Ideally, an example should be added to the examples folder.

There should be a subsection in this documentation page to warn about differences in PUB-SUB when using raw sockets (i.e.: topic is not separated from the message, the user must take care of that...). It just makes sense to have a "raw" ZeroMQ behavior when using the raw serialization.

Module core.py only contains agent-related logic, so it should be better renamed to agent.py before 1.0.0 release.

• Test handler with 2 parameters (agent, message)
• Test handler with 3 parameters (agent, message, topic)
• Test topic is properly filtered (no match, partial match, full match)

Document #30.

Implemented:

• proxy.safe.<methodcall> and proxy.unsafe.<methodcall> for per-call configuration.
• Proxy(..., safe=True/False) initialization parameter.
• Global OSBRAIN_DEFAULT_SAFE environment variable.

Set Dill as the default serializer for proxies.

Instead of manually executing .send(...) to reply earlier, could we use yield?

• BaseAgent --> Agent (keep "class BaseAgent(Agent): pass" for a while?)
• Agent --> AgentProcess
• run_agent(..., base=...) --> run_agent(..., agent=...)

https://travis-ci.org/opensistemas-hub/osbrain/jobs/212115613

Saw that before also for Python 3.3 version. In Python 3.4 that was fixed forcing a higher minor version. Try to force a higher minor Python 3.3.X version...

Timers can reach a maximum recursion level and throw an exception.

To avoid REQ-REP blocking, which would be useful for non-blocking safe calls, PUB-SUB synchronization, etc..

The documentation is up and running in readthedocs, but all the links/references to the documentation (i.e.: github, readme...) should be updated.

Old documentation from pypi (pythonhosted) should be removed.

https://travis-ci.org/opensistemas-hub/osbrain/jobs/212038883
https://travis-ci.org/opensistemas-hub/osbrain/jobs/212115610

Currently there is no way to close timers from proxies.

Check possible addition of type hints and use of Mypy within osBrain.

Reach at least 95% coverage, including branch coverage.

The only file currently under 95% is core.py.

Integrate asyncio syntax in osBrain where possible? (it could be very convenient in some cases)

As of now, the default method for communication between agents is through pickle serialization. This makes feeding data through outside Python a difficult task. Use of picke should be optional. Add raw message passing?

Set which methods are to be called safe/unsafe by default with decorators.

The default, currently, is a multiplexed server. This, however, prevents the user from accessing concurrently an agent unless configured to use a threaded server before running it.

In order to allow fine-grained control, a threaded server approach would be better as long as, by default, proxies always used safe calls (serializing calls to the main thread using an inproc socket). With this approach, the user would be able to explicitly change the call to be unsafe and decide, for each call, between safe/unsafe.

Some calls like send could be async by default (i.e.: no return value).

Some calls could be changed to be unsafe by default (for example addr, or calls that are only reading memory).

• Depends on #30.

Depends on #59

Perhaps we could use https://github.com/giampaolo/psutil

import psutil
psutil.pids()

[1, 2, 3, 4, 5, 6, 7, 46, 48, 50, 51, 178, 182, 222, 223, 224,
268, 1215, 1216, 1220, 1221, 1243, 1244, 1301, 1601, 2237, 2355,
2637, 2774, 3932, 4176, 4177, 4185, 4187, 4189, 4225, 4243, 4245, 
4263, 4282, 4306, 4311, 4312, 4313, 4314, 4337, 4339, 4357, 4358, 
4363, 4383, 4395, 4408, 4433, 4443, 4445, 4446, 5167, 5234, 5235, 
5252, 5318, 5424, 5644, 6987, 7054, 7055, 7071]

Perhaps JSON should be added (included in Python standard libraries) and others could be considered such as MessagePack.

These two lines are a bit annoying:

https://codecov.io/gh/opensistemas-hub/osbrain/compare/35a869e754ab377db51c831415d2af34e7cb27a2...338b8b488bc9d1a0b6df366550535f3919b1272e/changes

Depending on Travis mood, they might be tested or they might not. We should create a test to ensure we cover those lines. Otherwise Codecov may report an incorrect decrease in test coverage (i.e.: as in #63).

Part II:

• Dill serialization.
• Timers.
• Shutting down systems.
• Accessing agents from another terminal through proxies.

#22

Implement a run_agent method in the name server.

Some name server methods are not accessible from a proxy. This should be fixed and a separate file for name server tests should be created.

osBrain currently works well with Pyro4-4.45. It should be updated to work with the latest release. Relevant information:

irmen/Pyro4@9465d23#diff-ef64357c95c4e8298a468307da6ffaa3R132

Make all safe calls async (try to call _loopback to see how it currently blocks: second message to inproc never gets processed...).

Depends on #30, #31.

For better performance, memoryview should be used to split subject and content in messages. Perhaps there are other parts in the code where this could be used (?).

Name server does not capture SIGINT.

Make the (default) transport protocol configurable in osBrain. Probably socket addresses should integrate information about the transport protocol (i.e.: a connect should always use the appropriate transport protocol).

Make osBrain work well with latest Pyro4 version.