A framework for creating services backed by Redis streams. The original design idea was for chat bots, but it has evolved into a very flexible execution platform.
this repo is not usable as-is for running botcannon, see main project template:
version: 0.2.0
messages are collected and placed into a stream where a minimum of 1 worker is available at all times. work for each message is executed in it's own process.
normal operation creates a 'hot' worker process along with a new message to ensure either the 'idle' or 'hot' worker processes pick up the message quickly. an optional 'lazy' flag waits for the previous process/task to complete before creating a new 'idle' worker process.
a botcannon service contains a collector
, a shell
and tasks
(optional). together these collectively form a chat bot, generic data collector, scheduler, listener, etc.
message data has to originate from somewhere. collector class uses a .read()
method that yields dicts. (chat clients or other data generation)
your code to interact with collected data. can become a chatbot's root (see torch
) for process exec or a service menu for a specific collector.
the default action with no tasks places a message in the stream and moves on. tasks are methods for the worker processes to execute on the message.
these optional modifiers/actions are initialized at startup for the workers (be mindful of connections).TASKCLASS.task(message, results)
is called for each task in order as listed in your yml config. where message
is the original message hash and results
is dict of values that can be over written by the subsequent tasks.
torch
is a special task that runs an input command against your
shell
usingfire
. see examples for more.
similar to a callback, a taskback message allows a worker process to relay a message back to the original process. good for gaining access to the session/process of the collector.
botcannon requires an ./app/
dir for the botcannon python files. botcannon loads files in this folder similarly to the python import
builtin.
yml config is easy
services:
SERVICE_NAME:
collector:
COLLECTOR_SPEC
shell:
SHELL_SPEC
tasks:
- TASK_SPECS1
- TASK_SPECS2
with SERVICE_NAME
being a name of your choosing and TASK_SPECS
being a list of specs.
. you will refer to you SERVICE_NAME
later when starting the service.
COLLECTOR_SPEC
, SHELL_SPEC
and individual TASK_SPECS
:
file: filename
entry: ClassNameInFile
conf:
key: config_name
kwargs:
firstkwarg: value1
secondkwarg: value2
file
is the python filename minus .py
extension. entry
is the class to be used as an entrypoint into the script.
conf:key
is the key name that will be used in redis for the kwargs
. if you declared config_name
at the top of your yml config, config_name
could be used without specifying the kwargs.
botcannon takes your entry
, kwargs
and applies them like so:
ClassNameInFile(**kwargs)
or
ClassNameInFile(firstkwarg=value1, secondkwarg=value2
now, depending on if the spec is a collector
or task
, there are specific implementations required for your ClassNameInFile
in order for botcannon to use them.
collector.read()
is called in a loop repeatedly and should be a method that yields items in an efficient, timely manner. the collector is single threaded and the state of the collector is reflected in the workers when using ./botcannon up SERVICE
. if the collector crashes, workers will be terminated too (there can be no work with no data coming in).
items yielded must be a dict with only strings for values and are written to the service's stream.
the message schema from your collector is up to you, generally speaking try to mimic slack/rocketchat for new chat clients. if you need to, you can encode/decode some json to make bulk data available to workers.
if there are tasks for the service and those tasks have results, the worker will write to a taskback stream and the original collector process will pick up up that message, allowing actions to be called back to the original collector process, calling collector.taskback(message)
after completing a collection loop, if a message is present.
a shell
has no schema required other than being a class. this class can be some kind of tooling in the case of a chatbot, or it can be something like the roombot
demo (see template) where it uses a different service's data to expand the capability of the overall botcannon environment
multiple services in your cannon-compose.yml files are encouraged, as long as they are supporting each other. new templates should be made for new eco systems (dev, prod, qa, ops, dev-ci, etc).
you should have a good grasp of what a task
does from the collector
. tasks
determine if a taskback
is called and is where any significant computation/io/wait should take place, as this will happen in a worker process.
tasks
have full access to each message and the results of the previous task.