Multiple ExecutionCompletedEvent found about durabletask-mssql HOT 9 OPEN

Unfortunately, no updates yet. I was on a long vacation and apparently our GitHub automation which is supposed to help us track issues stopped working in the past few months. I'll make sure that this shows up in our triage so that we can get someone assigned to take a closer look.

from durabletask-mssql.

@bhugot: I'm working on a fix for this issue here - Azure/durabletask#949

from durabletask-mssql.

This means that multiple "completion" events have been found in your orchestration history state. The code that raises this error is here. You can see the full set of events here. The "completion" events are ExecutionCompleted and ExecutionTerminated.

To understand why you're seeing this, we'd need to look at the union of your History and NewEvents tables, filtered by your orchestration instance ID, to see whether/why there are multiple completion events for your orchestration that are triggering this error.

from durabletask-mssql.

Here are some data

from durabletask-mssql.

And I compared to an equivalent sucessfully completed orchestration. We have exactly the same history It's look like the event is registered but not deleted

from durabletask-mssql.

@cgillum any news on this?

from durabletask-mssql.

For information I think the use case is a functions for some reason is running twice (no retry configured). In the 2 case i met one was not idempotent and the second run failed. In the other case it was idempotent so both were successful

from durabletask-mssql.

New error on same subject I got

TaskOrchestrationDispatcher-b8699d665c904134b8e019ba498238dc-0: Unhandled exception with work item '413826bf-e213-4cca-a67a-77c495889fbe': System.InvalidCastException: Unable to cast object of type 'DurableTask.Core.History.TaskCompletedEvent' to type 'DurableTask.Core.History.TaskScheduledEvent'.
   at DurableTask.Core.TaskOrchestrationDispatcher.ReconcileMessagesWithState(TaskOrchestrationWorkItem workItem) in /_/src/DurableTask.Core/TaskOrchestrationDispatcher.cs:line 837
   at DurableTask.Core.TaskOrchestrationDispatcher.OnProcessWorkItemAsync(TaskOrchestrationWorkItem workItem) in /_/src/DurableTask.Core/TaskOrchestrationDispatcher.cs:line 328
   at DurableTask.Core.TaskOrchestrationDispatcher.OnProcessWorkItemAsync(TaskOrchestrationWorkItem workItem)
   at DurableTask.Core.TaskOrchestrationDispatcher.OnProcessWorkItemSessionAsync(TaskOrchestrationWorkItem workItem) in /_/src/DurableTask.Core/TaskOrchestrationDispatcher.cs:line 194
   at DurableTask.Core.WorkItemDispatcher`1.ProcessWorkItemAsync(WorkItemDispatcherContext context, Object workItemObj) in /_/src/DurableTask.Core/WorkItemDispatcher.cs:line 373

Backing off for 1 seconds until 5 successful operations

from durabletask-mssql.

And this time I had 2 task who are sequential in TaskCompleted Status

from durabletask-mssql.