Comments (7)
I had thought that Netherite's implementation of Event Hub in combination with FASTER would make it unlikely that such multiple execution of Activity would occur, but it seems that it is occurring more than I had expected.
That is not surprising. Duplicate activity executions are considered a normal occurrence. The most common reason for duplicate activity executions is when partitions are moved for load balancing reasons. The partition manager is not being careful about this, i.e. it will not hesitate to move a partition to a different host even if it means that such movement creates duplicate activity executions.
from durabletask-netherite.
Hi @shibayan, thanks for reaching out.
It's a bit difficult to diagnose this issue without having more context. Would you be able to provide us with an example instanceID that experienced this issue, the activity name that got doubly executed, the app name on Azure, and the timerange in UTC where this issue can be observed? From there, we should be able to assist better. Thanks!
from durabletask-netherite.
@davidmrdavid Thanks for the reply. Since this issue occurred in a production environment, I would like to exchange App Service information and other information via email. Can you provide me with an email address?
from durabletask-netherite.
Thanks @shibayan.
Agreed, this isn't entirely surprising and duplicate activity executions are allowed by the framework (that's why activities need to be idempotent), they're just not supposed to happen very often.
If you still would like me to take a peek, you can DM me on twitter (https://twitter.com/davidjustodavid) and I'll send you my work email there (just trying to avoid making it public on GitHub). But yeah unless this is happening extremely frequently, this may very well be part of normal operation for the framework.
from durabletask-netherite.
@sebastianburckhardt @davidmrdavid Thank you for your reply, I understand that the multiple execution of Activities is an acceptable behavior by design.
This design behavior may not be well known by Durable Functions users. The documentation says that non-deterministic operations should be performed in Activity, so it seems that there are many cases of misunderstanding like mine.
Depending on which language you use, a built-in API for generating deterministic GUIDs or UUIDs may be available. Otherwise, use an activity function to return a randomly generated GUID or UUID.
from durabletask-netherite.
Thanks @shibayan. I'm trying to understand how to make this guidance clearer. You're correct that Activities are where non-deterministic operations should be performed. At the same time, activities may execute more than once. What the Durable Functions framework does in this case is that, as soon as one activity has its output durably stored, the activity should no longer execute. Therefore, even though the activity may execute more than once, only one of it's results will be acknowledged by the orchestrator and therefore it will be as-if it had only executed once from the perspective of the workflow itself.
Does that makes sense? Do you have any suggestions on how to rephrase our guidance so that this can be made clearer? Thanks!
from durabletask-netherite.
@davidmrdavid It makes sense. My customer really likes Durable Functions/Netherite and uses them extensively, so I wanted to make the behavior clear.
The question was successfully resolved. I would like to provide feedback on the documentation separately. Thanks!
from durabletask-netherite.
Related Issues (20)
- Netherite scaling not working on consumption plan HOT 1
- Shadow store support HOT 2
- Entities become unresponsive under load HOT 7
- Partitions become unresponsive HOT 7
- partitions become unresponsive because recovery hangs after reading object log HOT 5
- Startup error : Storage provider type (Netherite) was not found HOT 19
- NetheriteOrchestrationService.StartAsync throws cached exception on retry if first call fails
- Question about performance warning logs HOT 4
- Unable to Debug Durable Entities on local machine HOT 2
- Partition becomes unresponsive because event batch blob was deleted prematurely HOT 5
- Question: is there support for the Event Hub to have private endpoints? HOT 2
- overwrite amqp protocol to amqpWebSockets HOT 1
- Purging and getting instances with DurableTaskClient in dotnet-isolated runtime HOT 3
- Large amounts of warnings from FASTER.KV HOT 13
- Perhaps longer latencies for Netherite for medium-sized payloads HOT 13
- System.NotSupportedException: Durable entities are not supported by the current backend configuration HOT 16
- Activity Function started multiple times before completion HOT 1
- Duplicate Messages from SignalEntityAsync HOT 2
- Error in orchestrator: Netherite backend failed to start: Operations per second is over the account limit. HOT 9
- ScheduleNewOrchestrationInstanceAsync fails when scale controller scaling in HOT 10
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from durabletask-netherite.