Comments (6)
Hi,
This depends on your configurations, there's a few things to keep in mind:
- Idle Queue Retention Period:
withIdleQueueRetentionPeriodSeconds(long period)
, sets the retention period of the virtual queue even if idle. - Idle Queue Sweeping Period:
withIdleQueueSweepingPeriod(int period, TimeUnit timeUnit)
, sets the sweeping period to check for all queues starting with for idleness and deletes them. - Queue Heartbeat Interval:
withQueueHeartbeatInterval(long heartbeatIntervalSeconds)
, sets the interval for the heartbeating mechanism to keep the queue alive.
When creating a new Requester Client, the default value for the idle queue sweeping period is of 5 minutes. This means that every 5 minutes it will check for all queues beginning with the internal queue prefix for idleness. It checks 2 things:
- Is the queue Idle: Checks whether (current timestamp - the last heart beat timestamp) is greater than the idle queue retention period
- Is the queue empty
If both those cases are true, then the idle queue will get deleted.
Note: A Host queue will not get deleted by the idle queue sweeper
By default, the heart beating mechanism will tag the queue every 5 seconds, the idle queue retention period is set to 5 minutes and the idle queue sweeping period is set to 5 minutes. This means that the queue will never get deleted even if it is idle, since the heart beat is keeping it alive.
sqsRequester = AmazonSQSRequesterClientBuilder.standard()
.withAmazonSQS(sqs)
.withIdleQueueSweepingPeriod(60, TimeUnit.SECONDS)
.withQueueHeartbeatInterval(420)
.build();
In this example, we set the idle queue sweeping period to every minute, meaning it will check for idle queues every minute. The heart beat is set to every 7 minutes and by default the idle queue retention period is set to 5 minutes. Therefore on the 6th minute, if a virtual queue has been idle, it will get deleted by the idle queue sweeper. On send and receive message calls, the heart beat timestamp is updated, allowing the virtual queue to stay alive.
from amazon-sqs-java-temporary-queues-client.
Hi @adam-aws , what do you mean? I've got the configuration by default (heartbeat=5 sc, idleQueueRetention=5 min, idleQueueSweeping=5 min). With the configuration by default, all the temporary queues will never be idle? Which is the recommended configuration?
from amazon-sqs-java-temporary-queues-client.
The number of Temporary queues continues to increase, and on application client, we have a lot of log messages:
[ERROR] 2021-06-25 13:38:58,378 SQSQueueUtils - Unexpected exception com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1207) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1153) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530) at com.amazonaws.services.sqs.AmazonSQSClient.doInvoke(AmazonSQSClient.java:2243) at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2210) at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2199) at com.amazonaws.services.sqs.AmazonSQSClient.executeTagQueue(AmazonSQSClient.java:2087) at com.amazonaws.services.sqs.AmazonSQSClient.tagQueue(AmazonSQSClient.java:2058) at com.amazonaws.services.sqs.AmazonSQSClient.tagQueue(AmazonSQSClient.java:2099) at com.amazonaws.services.sqs.AmazonSQSIdleQueueDeletingClient.heartbeatToQueue(AmazonSQSIdleQueueDeletingClient.java:281) at com.amazonaws.services.sqs.AmazonSQSIdleQueueDeletingClient.heartbeatToQueueIfNecessary(AmazonSQSIdleQueueDeletingClient.java:297) at com.amazonaws.services.sqs.AmazonSQSIdleQueueDeletingClient.receiveMessage(AmazonSQSIdleQueueDeletingClient.java:381) at com.amazonaws.services.sqs.AmazonSQSVirtualQueuesClient.lambda$receiveMessage$5(AmazonSQSVirtualQueuesClient.java:204) at java.util.Optional.orElseGet(Optional.java:267) at com.amazonaws.services.sqs.AmazonSQSVirtualQueuesClient.receiveMessage(AmazonSQSVirtualQueuesClient.java:204) at com.amazonaws.services.sqs.util.SQSMessageConsumer.poll(SQSMessageConsumer.java:118) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:286) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:263) at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1333) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145) ... 23 more [ERROR] 2021-06-25 13:38:58,420 SQSQueueUtils - Unexpected exception com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1207) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1153) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530) at com.amazonaws.services.sqs.AmazonSQSClient.doInvoke(AmazonSQSClient.java:2243) at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2210) at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2199) at com.amazonaws.services.sqs.AmazonSQSClient.executeReceiveMessage(AmazonSQSClient.java:1637) at com.amazonaws.services.sqs.AmazonSQSClient.receiveMessage(AmazonSQSClient.java:1607) at com.amazonaws.services.sqs.util.AbstractAmazonSQSClientWrapper.receiveMessage(AbstractAmazonSQSClientWrapper.java:153) at com.amazonaws.services.sqs.AmazonSQSIdleQueueDeletingClient.receiveMessage(AmazonSQSIdleQueueDeletingClient.java:382) at com.amazonaws.services.sqs.AmazonSQSVirtualQueuesClient.lambda$receiveMessage$5(AmazonSQSVirtualQueuesClient.java:204) at java.util.Optional.orElseGet(Optional.java:267) at com.amazonaws.services.sqs.AmazonSQSVirtualQueuesClient.receiveMessage(AmazonSQSVirtualQueuesClient.java:204) at com.amazonaws.services.sqs.util.AbstractAmazonSQSClientWrapper.receiveMessage(AbstractAmazonSQSClientWrapper.java:153) at com.amazonaws.services.sqs.util.SQSMessageConsumer.poll(SQSMessageConsumer.java:118) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:286) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:263) at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1333) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145) ... 22 more
from amazon-sqs-java-temporary-queues-client.
Can you try and update the configurations so that it fits your needs? So increasing the heartbeat interval to not have it do it every 5 seconds, since the sweeper runs every 5mins it wont get deleted.
Also, a queue is only βidleβ and automatically deleted if it is empty and has no traffic, including empty receives.
from amazon-sqs-java-temporary-queues-client.
@adam-aws , can you say me which is the recommended configuration? The default configuration makes SQS temporary queues to not work? If I increase the heartbeat from 5 seconds, what is the good value? any value greater than idle queue sweeper?
And what about the exception that appears on my last comment?
[ERROR] 2021-06-25 13:38:58,378 SQSQueueUtils - Unexpected exception com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1207) at
from amazon-sqs-java-temporary-queues-client.
Yes correct, so you can configure the Idle Queue Retention Period if you want to make sure the queue stays alive for a certain time, even if idle (by default 5min). Then based on your needs, you can set the idle queue sweepers value (by default 5min) if you want the sweeper to check quicker or slower.
You can change the heartbeat to be longer than the sweeper, otherwise, the queue will never be considered idle and therefore never deleted.
Concerning the errors, seems like your application is using too many threads which is exceeding the AWS SDK limit.
from amazon-sqs-java-temporary-queues-client.
Related Issues (20)
- Setup CI for Github project
- Upgrade to JUnit 5
- Implement the async version of the SQS API HOT 1
- Bug receiveMessage on a virtual queue only heartbeats at start of call
- Max Idle Queue Retention Period is set to 5 minutes HOT 1
- Reduce TPS to tagging actions in AmazonSQSIdleQueueDeletingClient HOT 1
- Separate request/response queues? HOT 1
- Orphaned message warnings HOT 1
- SQS tag length exceeded
- Failover SQS client
- Ability to create virtual queues with custom Tags HOT 1
- IdleQueueSweeper with incorrect log classname HOT 1
- SQSMessageConsumer causes ForkJoinPool.commonPool-worker java.security.AccessControlException
- Describe mechanism used when receiving response
- Snyk Vulnerabilities HOT 1
- AWS Temporary Queues support for . Net (C#) HOT 1
- Virtual queue cannot subscribe to SNS topic
- messages are not processed
- Unexpected exception during execution of AmazonSQSRequester sendMessageAndGetResponse
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from amazon-sqs-java-temporary-queues-client.