codingjoe / joeflow Goto Github PK

View Code? Open in Web Editor NEW

78.0 9.0 20.0 423 KB

The workflow automation framework for machines with heart

Home Page: https://joeflow.rtfd.io

License: BSD 3-Clause "New" or "Revised" License

Python 97.92% HTML 2.08%

workflow workflow-engine framework python django process automation edges automation-framework

joeflow's Introduction

joeflow

The lean workflow automation framework for machines with heart.

Joeflow is a free workflow automation framework designed to bring simplicity to complex workflows. Joeflow written in Python based on the world famous Django web framework.

Here is a little sample of what a workflow written in joeflow may look like:

from django.core.mail import send_mail
from jowflow.models import WorkflowState
from joeflow import tasks
from joeflow.workflows import Workflow


class Shipment(WorkflowState):
    email = models.EmailField(blank=True)
    shipping_address = models.TextField()
    tracking_code = models.TextField()

class ShippingWorkflow(Shipment, Workflow):
    checkout = tasks.StartView(fields=["shipping_address", "email"])

    ship = tasks.UpdateView(fields=["tracking_code"])

    def has_email(self):
        if self.email:
            return [self.send_tracking_code]

    def send_tracking_code(self):
        send_mail(
            subject="Your tracking code",
            message=self.tracking_code,
            from_email=None,
            recipient_list=[self.email],
        )

    def end(self, task):
        pass

    edges = [
        (checkout, ship),
        (ship, has_email),
        (has_email, send_tracking_code),
        (has_email, end),
        (send_tracking_code, end),
    ]

Design Principles

Common sense is better than convention

Joeflow does not follow any academic modeling notation developed by a poor PhD student who actually never worked a day in their life. Businesses are already complex which is why Joeflow is rather simple. There are only two types of tasks – human & machine – as well as edges to connect them. It’s so simple a toddler (or your CEO) could design a workflow.

Lean Automation (breaking the rules)

Things don’t always go according to plan especially when humans are involved. Even the best workflow can’t cover all possible edge cases. Joeflow embraces that fact. It allows uses to interrupt a workflow at any given point and modify it’s current state. All while tracking all changes. This allows developers to automate the main cases and users handle manually exceptions. This allows you businesses to ship prototypes and MVPs of workflows. Improvements can be shipped in multiple iterations without disrupting the business.

People

Joeflow is build with all users in mind. Managers should be able to develop better workflows. Users should able to interact with the tasks every single day. And developers should be able to rapidly develop and test new features.

Free

Joeflow is open source and collaboratively developed by industry leaders in automation and digital innovation.

Photo by rawpixel.com from Pexels

joeflow's People

Contributors

Stargazers

Watchers

Forkers

zhangyucumt jordotech jacques-cadet sebastiankapunkt andrewhyeon-jahnelgroup tubbz-alt wingi11 buzzit-jimmytse glebiuss buildforbetterworld marcocapozzoli lgtm-migrator dgreisen mohsenjavidpanah ccurvey peterbaumert yuanxiaoming8899 jbimat fermankamil

joeflow's Issues

Task assignee issue

I am currently adding assignees to a Task object via a post_save signal by matching the task.name against a string. Is it possible to add an assignee from within a workflow method?

Machine tasks are never finished

Greetings, @codingjoe and thanks for the great lib!

I tried to implement a flow that starts with the machine task.
Here is a close example of what I have:

from django.db import models
from joeflow.models import Workflow
from joeflow import tasks


class StartWorkflow(Workflow):
    start = tasks.Start()

    def end(self):
        pass

    edges = (
        (start, end),
    )

workflow = StartWorkflow.start()

When I execute this workflow, start task is scheduled, then it triggers end, which is then becomes finished.
However, the status of start task never changes and remains as scheduled.

I briefly checked the code and could not find where in the machine-generated task method finished is called like it is done in TaskViewMixin.next_task.

I will try to dig into it and provide a test case with a patch to this bug.

Best,
Rust

Functionnal demo examples in the code

Could it be possible to add a "Demo" directory in the code to test quickly that library.
That directory could contain :

The simple examples explain in the documentation : ShippingWorkflow, WelcomeWorkflow, SimpleWorkflow, GatewayWorkflow, SplitJoinWorkflow, LoopWorkflow, WaitWorkflow
A full complete example extracted from real life

Graphs not rendering on ReadTheDocs

To me it looks like somebody took care to add graphs, but they're not rendering on RTD. I am not sure where else I could browse the docs to have a look at the graph. I really want to see the graph. <3

Alternative (top down) rendering of the instance diagrams

I would like to show instance diagrams rendered with TD (top down) and not default LR (left right) directions.
In the current version I have to alter internal joeflow code.
The obvious way would be to add parameter, but this wouldn't work from django templates.

Graphviz 0.18 incompatibility

There was a recent Graphviz release, which seem to break this package.

https://github.com/xflr6/graphviz/blob/master/CHANGES.rst#version-018

After a quick check, I see at least one issue:
xflr6/graphviz@17df613 changed behaviour of private _head and _head_strict attributes - they are static methods now.

I am happy to help, but not familiar with the codebase.
Nevertheless, I guess they way to go would be:

Pin graphviz to <0.18 and release a minor/bugfix version
Fix incompatibility and release a new major version

What do you think?

how to activate joeflow admin interface?

There is the admin.py inside the joeflow, but it doesn't show up under /admin url
Very likely I am missing something obvious in configuration.
What is the way to activate the predefined joeflow/admin.py ?

Usage in App is not working because of namespace issues

https://github.com/codingjoe/joeflow/blob/8d58695eeee2502d731fb5dde2491dba63e28c09/joeflow/models.py#L177C29-L177C29

If I use joeflow in an app in my project, the method to get the url namespace returns a wrong resulst.

it should be something like

return f"{cls._meta.app_label}:{cls.__name__.lower()}"

maybe somehow conditionally check if cls._meta.app_label exists

additionally one needs to specify the urls as follows:

path(
        "whateveriwant/",
        include(workflows.VeryCoolWorkflow.urls(), namespace="verycoolworkflow"),
    ),

Notes on first time working with the package

1) Workflow & State

For me the separation between a workflow and state is not quite obvious in the first place.
Also, that a state is actually a new model is not obvious. (In the Tutorial it is not clear why the flow or the state needs a User)

2) Entry point of a Task

In the tutorial the start edge is defined as variable on the workflow. But it is unclear what all kinds of implications this has. Once you have reached all parts connected to the start edge it gets more clear.

3) Workflow creation

For me, it took some time to understand that for any workflow you want to create, you actually have to create an object. At first glance it could be like an add-on where you hook up to the model itself and just add tasks. But now I understand that each workflow needs a model representation (and that it can be an existing model). And that migrations are needed.

For a bit more complicated use cases or integration with an existing model it would be nice to have some documentation.

All in all it is a really nice package and well-thought-out.

tasks.Wait make default retries of 180 seconds regardless of timedelta

Hey @codingjoe, congratulations on the library. Very nice work.

Could you tell me if there is a way to configure the task retry time?

While using it I've noticed that whenever a machine task returns False it will auto-retry within 180 seconds (for celery). Even if I make the flow goes with a tasks.Wait(timedelta(days=1)) for instance, the Wait task will return False and keep retrying every 180 seconds. So, tasks that should hang for long periods keep getting evaluated without need.

cannot run tutorial (stuck on "has user")

software versions:
Python 3.10.4
Django==4.0.5
joeflow==1.1.1
celery==5.2.7

Thanks for this library. We're investigating whether it is something we can use at our small nonprofit, but we've run into a blocker:

After setting celery up and running it in background, I tried creating a welcome workflow via 127.0.0.1:8000/welcome/start/. Detail view shows the manual start task successfully completed, but has user has a completed of None.

When I drop into shell I get:

>>> workflow = WelcomeWorkflow.objects.all()[1]
>>> has_user_task = workflow.task_set.all()[1]
>>> has_user_task.status
'scheduled'

I was worried it might be a celery issue, so I started with the celery example app (https://github.com/celery/celery/tree/master/examples/django/#running-a-task) made sure I could run the demo tasks, then added the joeflow example to that example app.

Any suggestions where to go from here? Thank you!

The following are issues found in the tutorial. Happy to submit a docs PR if that would save you some time.

detail view missing template resolution order found in other templates
"and add joeflow to the INSTALLED_APP" is a very important bit lost between the two large grey boxes that draw the eye.
missing that you need to update the settings to switch to celery backend.
tutorial missing that you need to install graphviz (https://graphviz.org/download/).
I would suggest moving dependencies into an unordered list
would appreciate the filename of each code block to help orientate and provide context. e.g., first two code blocks would be workflows.py? I think?
Would be great to have some troubleshooting tips:
- if get circular import, then likely forgot to add joeflow to INSTALLED_APPS

ValueError: Field 'id' expected a number but got a string

The following error occurs if someone tries to access a URL workflow detail view with a non integer primary key in the URL.
We should adjust the URL pattern to only accept integers to avoid this error and raise the expected 404 response.

ValueError: invalid literal for int() with base 10: 'some-string'
  File "django/db/models/fields/__init__.py", line 1774, in get_prep_value
    return int(value)
ValueError: Field 'id' expected a number but got 'some-string'.
  File "django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "newrelic/hooks/framework_django.py", line 554, in wrapper
    return wrapped(*args, **kwargs)
  File "django/views/generic/base.py", line 70, in view
    return self.dispatch(request, *args, **kwargs)
  File "newrelic/hooks/framework_django.py", line 944, in wrapper
    return wrapped(*args, **kwargs)
  File "django/views/generic/base.py", line 98, in dispatch
    return handler(request, *args, **kwargs)
  File "django/views/generic/detail.py", line 106, in get
    self.object = self.get_object()
  File "django/views/generic/detail.py", line 36, in get_object
    queryset = queryset.filter(pk=pk)
  File "django/db/models/query.py", line 942, in filter
    return self._filter_or_exclude(False, *args, **kwargs)
  File "django/db/models/query.py", line 962, in _filter_or_exclude
    clone._filter_or_exclude_inplace(negate, *args, **kwargs)
  File "django/db/models/query.py", line 969, in _filter_or_exclude_inplace
    self._query.add_q(Q(*args, **kwargs))
  File "django/db/models/sql/query.py", line 1358, in add_q
    clause, _ = self._add_q(q_object, self.used_aliases)
  File "django/db/models/sql/query.py", line 1377, in _add_q
    child_clause, needed_inner = self.build_filter(
  File "django/db/models/sql/query.py", line 1319, in build_filter
    condition = self.build_lookup(lookups, col, value)
  File "django/db/models/sql/query.py", line 1165, in build_lookup
    lookup = lookup_class(lhs, rhs)
  File "django/db/models/lookups.py", line 24, in __init__
    self.rhs = self.get_prep_lookup()
  File "django/db/models/fields/related_lookups.py", line 117, in get_prep_lookup
    self.rhs = target_field.get_prep_value(self.rhs)
  File "django/db/models/fields/__init__.py", line 1776, in get_prep_value
    raise e.__class__(

Circular import error

"D:\python_project\PyCharm 2021.1.1\bin\runnerw64.exe" D:\python_project\view_flow_project\venv\Scripts\python.exe D:/python_project/view_flow_project/manage.py runserver 8000
Watching for file changes with StatReloader
Exception in thread django-main-thread:
Traceback (most recent call last):
  File "C:\Users\Bogdan\AppData\Local\Programs\Python\Python36\lib\threading.py", line 916, in _bootstrap_inner
    self.run()
  File "C:\Users\Bogdan\AppData\Local\Programs\Python\Python36\lib\threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "D:\python_project\view_flow_project\venv\lib\site-packages\django\utils\autoreload.py", line 64, in wrapper
    fn(*args, **kwargs)
  File "D:\python_project\view_flow_project\venv\lib\site-packages\django\core\management\commands\runserver.py", line 110, in inner_run
    autoreload.raise_last_exception()
  File "D:\python_project\view_flow_project\venv\lib\site-packages\django\utils\autoreload.py", line 87, in raise_last_exception
    raise _exception[1]
  File "D:\python_project\view_flow_project\venv\lib\site-packages\django\core\management\__init__.py", line 375, in execute
    autoreload.check_errors(django.setup)()
  File "D:\python_project\view_flow_project\venv\lib\site-packages\django\utils\autoreload.py", line 64, in wrapper
    fn(*args, **kwargs)
  File "D:\python_project\view_flow_project\venv\lib\site-packages\django\__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "D:\python_project\view_flow_project\venv\lib\site-packages\django\apps\registry.py", line 114, in populate
    app_config.import_models()
  File "D:\python_project\view_flow_project\venv\lib\site-packages\django\apps\config.py", line 301, in import_models
    self.models_module = import_module(models_module_name)
  File "C:\Users\Bogdan\AppData\Local\Programs\Python\Python36\lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "D:\python_project\view_flow_project\view_flow\models.py", line 7, in <module>
    from joeflow import tasks
  File "D:\python_project\view_flow_project\venv\lib\site-packages\joeflow\tasks\__init__.py", line 7, in <module>
    from .human import *  # NoQA
  File "D:\python_project\view_flow_project\venv\lib\site-packages\joeflow\tasks\human.py", line 4, in <module>
    from joeflow.views import TaskViewMixin
  File "D:\python_project\view_flow_project\venv\lib\site-packages\joeflow\views.py", line 7, in <module>
    from . import forms, models
  File "D:\python_project\view_flow_project\venv\lib\site-packages\joeflow\forms.py", line 4, in <module>
    from . import models
  File "D:\python_project\view_flow_project\venv\lib\site-packages\joeflow\models.py", line 55, in <module>
    class Workflow(models.Model, metaclass=WorkflowBase):
  File "D:\python_project\view_flow_project\venv\lib\site-packages\joeflow\models.py", line 97, in Workflow
    override_view = views.OverrideView
AttributeError: module 'joeflow.views' has no attribute 'OverrideView'


Process finished with exit code 0

how can this be fixed? thanks

using function-based views

I'm a Djangosaur, and I haven't learned class-based views yet. Is there a way to use function-based views with joeflow?

(And please let me know if there's a better location to ask questions.)

default schema for task is not oracle friendly

makemigrations against oracle as db backend creates the following warnings

(venv) λ python manage.py makemigrations
System check identified some issues:

WARNINGS:
joeflow.Task.name: (fields.W162) Oracle does not support a database index on NCLOB columns.
        HINT: An index won't be created. Silence this warning if you don't care about it.
joeflow.Task.status: (fields.W162) Oracle does not support a database index on NCLOB columns.
        HINT: An index won't be created. Silence this warning if you don't care about it.
joeflow.Task.type: (fields.W162) Oracle does not support a database index on NCLOB columns.
        HINT: An index won't be created. Silence this warning if you don't care about it.

Would it make more sense to use CharFIeld instead of TextField for the 3 fields in question?
I'd prefer limited size fields with index than unlimited size fields without index.

locking depends on redis

This is more of the documentation issue. It is not clear that redis is a hard dependency.
As a workaround, I switched my celery to single worker and commented out active part of locking.py

next_nodes is sometimes [None] instead of None

https://github.com/codingjoe/joeflow/blob/8d58695eeee2502d731fb5dde2491dba63e28c09/joeflow/models.py#L564C31-L564C31

I don't know why but at me, sometimes next_nodes is [None] and i had to adjust the line to

if next_nodes is None or next_nodes == [None]:

overridden type (python keyword) with Task model field type

in Task model ( https://github.com/codingjoe/joeflow/blob/master/joeflow/models.py#L404 )
type is python keyword
with using same name it's shadowing keyword name.

so my preference is to update filed name like (task_type)
happy to raise PR for that

guide me if i'm wrong
thanks 😃