Comments (3)
thegrekle
commented on June 14, 2024
Here's the full stack trace in case that's helpful.
Error: The operation failed with an unexpected status code: 500. Details: {"Message":"Something went wrong while processing your request","ExceptionMessage":"Client request timed out.","ExceptionType":"System.TimeoutException","StackTrace":" at DurableTask.Netherite.Client.CreateTaskOrchestrationAsync(UInt32 partitionId, TaskMessage creationMessage, OrchestrationStatus[] dedupeStatuses) in /_/src/DurableTask.Netherite/OrchestrationService/Client.cs:line 469\r\n at DurableTask.Netherite.NetheriteOrchestrationService.DurableTask.Core.IOrchestrationServiceClient.CreateTaskOrchestrationAsync(TaskMessage creationMessage, OrchestrationStatus[] dedupeStatuses) in /_/src/DurableTask.Netherite/OrchestrationService/NetheriteOrchestrationService.cs:line 594\r\n at DurableTask.Core.TaskHubClient.InternalCreateOrchestrationInstanceWithRaisedEventAsync(String orchestrationName, String orchestrationVersion, String orchestrationInstanceId, Object orchestrationInput, IDictionary`2 orchestrationTags, OrchestrationStatus[] dedupeStatuses, String eventName, Object eventData, Nullable`1 startAt) in /_/src/DurableTask.Core/TaskHubClient.cs:line 614\r\n at Microsoft.Azure.WebJobs.Extensions.DurableTask.DurableClient.Microsoft.Azure.WebJobs.Extensions.DurableTask.IDurableOrchestrationClient.StartNewAsync[T](String orchestratorFunctionName, String instanceId, T input) in D:\\a\\_work\\1\\s\\src\\WebJobs.Extensions.DurableTask\\ContextImplementations\\DurableClient.cs:line 210\r\n at Microsoft.Azure.WebJobs.Extensions.DurableTask.HttpApiHandler.HandleStartOrchestratorRequestAsync(HttpRequestMessage request, String functionName, String instanceId) in D:\\a\\_work\\1\\s\\src\\WebJobs.Extensions.DurableTask\\HttpApiHandler.cs:line 875\r\n at Microsoft.Azure.WebJobs.Extensions.DurableTask.HttpApiHandler.HandleRequestAsync(HttpRequestMessage request) in D:\\a\\_work\\1\\s\\src\\WebJobs.Extensions.DurableTask\\HttpApiHandler.cs:line 306"}
at DurableOrchestrationClient.createGenericError (C:\home\site\wwwroot\node_modules\durable-functions\lib\src\durableorchestrationclient.js:612:16)
at DurableOrchestrationClient.<anonymous> (C:\home\site\wwwroot\node_modules\durable-functions\lib\src\durableorchestrationclient.js:404:44)
at Generator.next (<anonymous>)
at fulfilled (C:\home\site\wwwroot\node_modules\durable-functions\lib\src\durableorchestrationclient.js:5:58)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
from durabletask-netherite.
sebastianburckhardt
commented on June 14, 2024
Unfortunately, the client request timeouts can have many reasons. It basically means the client did not receive a response from the partition in time; but it contains no information about why. Usually the next step is to take a look at the logs for the target partition.
Is this running in a consumption or premium plan? If so, I can take a look at the trace in our internal telemetry if you tell me the appname.
from durabletask-netherite.
thegrekle
commented on June 14, 2024
Ended up working with Azure support on this. No real resolution, it appears that the task causes memory use on the function instances over time until these errors begin. It's reproduced several times in our environment now. Fresh task hub, we get good performance and no CPU/Memory warnings. Then after a few days the memory use increases to the point that performance drops and eventually leads to these client timeouts. Migrate to a new task hub and performance is good again.
Support just said we need to scale up or reduce memory consumption.
from durabletask-netherite.
Related Issues (20)
- Netherite scaling not working on consumption plan HOT 1
- Shadow store support HOT 2
- Entities become unresponsive under load HOT 7
- Partitions become unresponsive HOT 7
- partitions become unresponsive because recovery hangs after reading object log HOT 5
- Startup error : Storage provider type (Netherite) was not found HOT 19
- NetheriteOrchestrationService.StartAsync throws cached exception on retry if first call fails
- Question about performance warning logs HOT 4
- Unable to Debug Durable Entities on local machine HOT 2
- Partition becomes unresponsive because event batch blob was deleted prematurely HOT 5
- Question: is there support for the Event Hub to have private endpoints? HOT 2
- overwrite amqp protocol to amqpWebSockets HOT 1
- Purging and getting instances with DurableTaskClient in dotnet-isolated runtime HOT 3
- Large amounts of warnings from FASTER.KV HOT 13
- Perhaps longer latencies for Netherite for medium-sized payloads HOT 13
- System.NotSupportedException: Durable entities are not supported by the current backend configuration HOT 16
- Activity Function started multiple times before completion HOT 1
- Duplicate Messages from SignalEntityAsync HOT 2
- Error in orchestrator: Netherite backend failed to start: Operations per second is over the account limit. HOT 9
- ScheduleNewOrchestrationInstanceAsync fails when scale controller scaling in HOT 10
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from durabletask-netherite.