Task canceled exception about aws-sdk-net HOT 20 CLOSED

Lately, we are getting the exception below very often;

Amazon.Runtime.AmazonServiceException: A task was canceled. ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult() at Amazon.Runtime.AmazonWebServiceClient.<InvokeConfiguredRequest>d__31.MoveNext()
--- End of inner exception stack trace ---
at Amazon.Runtime.AmazonWebServiceClient.d__3`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Amazon.SQS.AmazonSQSClient.SendMessage(SendMessageRequest request)

from aws-sdk-net.

The methods internally use HttpClient.SendAsync method which can throw a TaskCanceledException in case of timeouts. This can be one reason that you are seeing this exception. We will fix this behavior so that the exception is wrapped properly. I'll update you when this fix is available.

Turning on logging can provide more information on why the operations failed. This link provides instructions on turning on logging for the SDK : http://blogs.aws.amazon.com/net/post/TxZLWAOFZJQWRP/Logging-with-the-AWS-SDK-for-NET
You can also enable response logging using the below configuration.

from aws-sdk-net.

That sounds good. I find it a bit unlikely that it was caused by a timeout, as we had a lot of exceptions on a limited timeframe. Could it be some resource that we are not disposing or something similar.

We are trying to reproduce the exception with logging enabled.

from aws-sdk-net.

After running a test to reproduce the issue for 3 days we have been unable to do so. We have been able to reproduce the issue by manually adding latency (using a tool called TMNetSim). Adding e.g. 20ms of delay will result in TaskCancelledException for all requests.

Is there any way we can harden the application to better be able to deal with these timeouts? E.g. Multiple clients vs. a single shared client, settings, etc?

from aws-sdk-net.

Hi, we have released a new version of the SDK (2.8.0.2) which wraps the TaskCanceledException with the AmazonServiceException for timeout scenarios. If you use the async methods and issue a cancellation you would still get either an OperationCanceledException or a TaskCanceledException (OperationCanceledException is the base class).

from aws-sdk-net.

I'll try to replicate the issue using the tool you mentioned.
The SDK uses default values for ReadWriteTimeout and Timeout properties. The default value for Timeout is 100 seconds (http://msdn.microsoft.com/en-us/library/system.net.http.httpclient.timeout%28v=vs.110%29.aspx) and the default value for ReadWriteTimeout is 300 seconds (http://msdn.microsoft.com/en-us/library/system.net.http.webrequesthandler.readwritetimeout%28v=vs.110%29.aspx). Currently only S3 and Glacier clients expose setting to change these timeouts via the AmazonGlacierConfig and AmazonS3Config classes.
I'm finding it difficult to understand how the 20ms delay is causing the timeouts. What is number of concurrent requests when you get these timeouts?

from aws-sdk-net.

Thank you for the release. I will upgrade and do further testing.

I was surprised about the low delay as well, which makes me wonder if there is some issue in our application. The test I did was pretty uncontrolled, and I do not know much about the tool I was using. I will try to look further into the issue and get back to if there is something that may be of value to you.

from aws-sdk-net.

I updated to 2.0.8.2 and we see that the exception has now been wrapped, but I was expecting to get an inner exception which explained the underlying issue. Instead I just get the TaskCancelledException which leaves me wondering what exactly caused the exception.

e.g.:

Exception: { Message = A task was canceled., Type = Amazon.Runtime.AmazonServiceException, StackTrace = at Amazon.DynamoDBv2.AmazonDynamoDBClient.UpdateItem(UpdateItemRequest request)
at QueueIT.ForecasterService.BusinessLogic.QueueIdWindow.PersistRedirectAttemptToDynamo(DateTime redirectTime, Int32 maxRedirects, TimeSpan openWindowSize) in d:\TFSData\QueueIT\Builds\SourceCode\ForecasterService\Test\Mainline\QueueIT.ForecasterService.BusinessLogic\QueueIdWindow.cs:line 154
at QueueIT.ForecasterService.BusinessLogic.QueueIdWindow.Redirect(Int32 maxRedirects, TimeSpan openWindowSize) in d:\TFSData\QueueIT\Builds\SourceCode\ForecasterService\Test\Mainline\QueueIT.ForecasterService.BusinessLogic\QueueIdWindow.cs:line 83
at QueueIT.ForecasterService.BusinessLogic.ForecasterEvent.TryRedirect(Guid queueId, DateTime inQueueTime) in d:\TFSData\QueueIT\Builds\SourceCode\ForecasterService\Test\Mainline\QueueIT.ForecasterService.BusinessLogic\ForecasterEvent.cs:line 343
at QueueIT.ForecasterService.ServiceInterface.Forecaster.TryRedirect(String customerId, String eventId, Guid queueId, DateTime inQueueTime) in d:\TFSData\QueueIT\Builds\SourceCode\ForecasterService\Test\Mainline\QueueIT.ForecasterService.ServiceInterface\Forecaster.cs:line 101, AmazonRequestId = }

InnerException: { Message = A task was canceled., Type = System.Threading.Tasks.TaskCanceledException, StackTrace = at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult() at Amazon.Runtime.AmazonWebServiceClient.<InvokeConfiguredRequest>d__31.MoveNext(), AmazonRequestId = }

from aws-sdk-net.

After looking into getting more detailed information about the root issue, it turns out that there is no more information (in inner exceptions or the Task object). So I guess this is by design by Microsoft - you are supposed to know that if there is a TaskCancelledException with IsCancellationRequested set to false it is a timeout. Bad design in my opinion but obviously not the fault of the AWS SDK.

Regarding the TMNetSim tool I was using - it was not really appropriate for the task. It used all of the CPU causing longer delays then what is was set to do.

from aws-sdk-net.

Yes, we are passing back the TaskCanceledException thrown by the HttpClient and it does populate the inner exception. This exception can be thrown in two cases by the HttpClient, timeout or user requested cancellation.
Detecting which case is not straight forward because the TaskCanceledException.CancellationToken.IsCancellationRequested is always set to false (in both scenarios), there is no way to set the CancellationToken property on the TaskCanceledException class as it does not provide a constructor which taken in a token.

In async scenarios, the way to detect this is to catch the TaskCanceledException and check the IsCancellationRequested on the original token (not on the exception) that you passed in to the async method in the first place. It's not an elegant solution.

I'm still interested to know the conditions which cause these exceptions to be thrown. Are you issuing requests under high load and that's making it queue up and timeout?

from aws-sdk-net.

We do not know the cause of the exception. The exceptions where thrown under load, but we have been running the test for 5 days now with higher load with no exceptions. With the information I have at this time I am assuming that is was a network issue. We have an open support case at AWS, but have no final explanation yet and I am not sure we will be able to determine the cause.

from aws-sdk-net.

Hi, the support case you opened is assigned to our team. It's difficult to figure out the root cause given the limited data and not being able to reproduce the issue. If you are able to repro the issue, you can open a new issue or support case, then we can investigate it further.

from aws-sdk-net.

I'm actually seeing the same issue - intermittent exceptions during the poll loop. I'm attaching a screenshot from our logs; I can find more detailed information if you like.

from aws-sdk-net.

Thanks for reporting this, can you provide more detailed information? This exception is thrown on request timeouts.
What version of the SDK are you using, what was the service call that threw the exception?

from aws-sdk-net.

We're using v4.0.30319, and the call was to PollForActivityTask.

On Thu, Feb 20, 2014 at 4:49 PM, Milind G [email protected] wrote:

Thanks for reporting this, can you provide more detailed information? This
exception is thrown on request timeouts.
What version of the SDK are you using, what was the service call that
threw the exception?

Reply to this email directly or view it on GitHubhttps://github.com//issues/67#issuecomment-35687846
.

from aws-sdk-net.

Thanks, can you post a sample code that reproduces this issue? Are you seeing this issue regularly?

from aws-sdk-net.

Can you let us know the version of the AWS SDK for .NET that you are using?

from aws-sdk-net.

I'm getting this error too. Happy to send you my code if that helps:
Server Error in '/' Application.

A task was canceled.

Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

Exception Details: System.Threading.Tasks.TaskCanceledException: A task was canceled.

Source Error:

Line 40: Key = key
Line 41: };
Line 42: using (GetObjectResponse oresponse = S3.s3client.GetObject(orequest))
Line 43: {
Line 44: filename=oresponse.Metadata["x-amz-meta-qqfilename"];

Source File: c:\websites\idtechex.com\htdocs\dotnet\s3\s3lib.cs Line: 42

Stack Trace:

[TaskCanceledException: A task was canceled.]
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +10484703
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +52
System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task) +10485256
Amazon.Runtime.d__3`1.MoveNext() +536

[AmazonServiceException: A task was canceled.]
Amazon.Runtime.d__3`1.MoveNext() +2982
Amazon.S3.AmazonS3Client.GetObject(GetObjectRequest request) +73
S3File..ctor(S3Object obj) in c:\websites\idtechex.com\htdocs\dotnet\s3\s3lib.cs:42
S3.getFiles(HttpServerUtility server) in c:\websites\idtechex.com\htdocs\dotnet\s3\s3lib.cs:197
ListFilesPage.Page_Load(Object sender, EventArgs e) in c:\websites\idtechex.com\htdocs\dotnet\s3\s3lib.cs:263
System.Web.Util.CalliEventHandlerDelegateProxy.Callback(Object sender, EventArgs e) +51
System.Web.UI.Control.OnLoad(EventArgs e) +92
System.Web.UI.Control.LoadRecursive() +54
System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +772

from aws-sdk-net.

Bumping this.. as we are experiencing seemingly random Task canceled exceptions calling DynamoDB on several different tables/indexes on our system.

» 20 Jan 2017 09:38:34.677 ->2017-01-20T09:38:32.7784060-06:00 microedge-services-tasks-4096662097-vu78r MicroEdge.Services.Tasks [Error] (System.Threading.Tasks.TaskCanceledException: A task was canceled. at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult() at System.Net.Http.HttpClient.<FinishSendAsync>d__58.MoveNext()--- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.Runtime.HttpWebRequestMessage.<GetResponseAsync>d__20.MoveNext()--- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.Runtime.Internal.HttpHandler`1.<InvokeAsync>d__9`1.MoveNext()--- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.Runtime.Internal.Unmarshaller.<InvokeAsync>d__3`1.MoveNext()--- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.Runtime.Internal.ErrorHandler.<InvokeAsync>d__5`1.MoveNext()--- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.Runtime.Internal.CallbackHandler.<InvokeAsync>d__9`1.MoveNext()--- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.Runtime.Internal.CredentialsRetriever.<InvokeAsync>d__7`1.MoveNext()--- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.Runtime.Internal.RetryHandler.<InvokeAsync>d__10`1.MoveNext()--- End of stack trace from previous location where exception was thrown --- at Amazon.Runtime.Internal.RetryHandler.<InvokeAsync>d__10`1.MoveNext()--- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.Runtime.Internal.CallbackHandler.<InvokeAsync>d__9`1.MoveNext()--- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.Runtime.Internal.CallbackHandler.<InvokeAsync>d__9`1.MoveNext()--- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.Runtime.Internal.ErrorCallbackHandler.<InvokeAsync>d__5`1.MoveNext()--- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.Runtime.Internal.MetricsHandler.<InvokeAsync>d__1`1.MoveNext()--- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult() at MicroEdge.Services.Tasks.Services.TasksService.<GetByAttributeIndex>d__6.MoveNext() in /app/Services/TasksService.cs:line 145--- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult() at MicroEdge.Services.Tasks.Services.TasksService.<GetByGrantId>d__8.MoveNext() in /app/Services/TasksService.cs:line 183--- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult() at MicroEdge.Services.Tasks.Services.TasksService.<Clear>d__11.MoveNext() in /app/Services/TasksService.cs:line 253--- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult() at MicroEdge.Services.Tasks.Controllers.TasksInternalController.<Clear>d__7.MoveNext() in /app/Controllers/TasksInternalController.cs:line 108--- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeActionFilterAsync>d__28.MoveNext()--- End of stack trace from previous location where exception was thrown --- at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeAsync>d__18.MoveNext()--- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AspNetCore.Builder.RouterMiddleware.<Invoke>d__4.MoveNext()--- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware`1.<Invoke>d__18.MoveNext()--- End of stack trace from previous location where exception was thrown --- at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware`1.<Invoke>d__18.MoveNext()--- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AspNetCore.Hosting.Internal.RequestServicesContainerMiddleware.<Invoke>d__3.MoveNext()--- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.Frame`1.<RequestProcessingAsync>d__2.MoveNe /

from aws-sdk-net.

For anyone stumbling over this year later. Please note that the timeout property is now exposed via the configuration objects of other services over and above s3 and glacier, so previously mentioned by @gokarnm. E.g. lambda: https://docs.aws.amazon.com/sdkfornet/v3/apidocs/items/Lambda/TLambdaConfig.html

from aws-sdk-net.