Better handling of OPEN_ACTIVITIES_LIMIT_EXCEEDED about simpleflow HOT 3 OPEN

OK, here's what happened in our workflow (simplified):

from simpleflow import Workflow

class OurWorkflow(Workflow):
    def run(self, **context):
        # a first activity that needs preparation (partitionning for instance)
        preparation = self.submit(prepare_expensive_activity)
        if preparation.finished:
            for i in range(0,400):
                self.submit(foo)

        # many other activities
        for i in range(0, 800):
            self.submit(bar)

        # wait for everybody to finish..
        # ...

Now what happens when playing this workflow from the decider point of view:

• replay 1:
  • 0 activity tasks open
  • preparation task is set to be scheduled,
  • 800 "bar" tasks are to be scheduled
  • => the decider sends a decision with 801 ScheduleActivityTask (in fact multiple decisions because of the 100-decision-tasks limit, but anyway)
  • => 801 activity tasks open
• replay 2:
  • (imagine) preparation task is finished, which triggered a new decision task
  • 800 activity tasks open
  • simpleflow enters the first conditional, and will try to submit the 400 activity tasks
  • => BOOM

Moral of the story: we cannot rely on the open activity tasks counter unless we parse the whole workflow. We'd have to defer tasks submission after we know exactly all tasks and their status.

Even then, some constructs could make simpleflow fail very easily, for instance if we bypass a conditional where we entered before and where one or many tasks are submitted.

=> option 1 looks a bit complex and we won't be able to implement a fully reliable solution if people mess with future.finished conditionals.

from simpleflow.

Instead of relying on a naive counter, I think we could rely on an array of open activities (probably just their "activityId"), so we don't start from zero but from this list. When replaying the workflow:

• start with the list of activities as seen in the history events
• when scheduling an activity: if it's in the list, don't do anything, if not, add it
• evaluate the length of the list as a replacement for the naive counter

This will protect against cases like the one described above (if done correctly). Now to plumb that together unless somebody has a better idea.

from simpleflow.

Option "6" (add a soft limit option) was added in fd22d8d and is available for simpleflow >= 0.10.2.

from simpleflow.