DNS resolution error when fetching more than 200 objects concurrently from Amazon S3 about aws-sdk-rust HOT 5 CLOSED

Hi @klefevre, thank you for reporting this. We've been able to reproduce the issue on our end. Will look into this further.

from aws-sdk-rust.

FYI, while we've reproduced it, it does not seem to be consistently reproducible. For instance, when I run the main function in aws-sdk-rust-concurrency-issue 4 times, with const CONCURRENCY_LIMIT: usize = 200 and const RANGE: Range<u32> = 1..2000 I get

1st run (success)
➜  aws-sdk-rust-concurrency-issue git:(main) ✗ cargo r        
    Finished dev [unoptimized + debuginfo] target(s) in 3.40s
     Running `target/debug/aws-sdk-rust-concurrent-issue`

2nd run (connection timeout)
➜  aws-sdk-rust-concurrency-issue git:(main) ✗ cargo r        
    Finished dev [unoptimized + debuginfo] target(s) in 0.10s
     Running `target/debug/aws-sdk-rust-concurrent-issue`
Error: Failed to collect objects

Caused by:
    0: Failed to fetch object 1
    1: dispatch failure
    2: timeout
    3: error trying to connect: HTTP connect timeout occurred after 3.1s
    4: HTTP connect timeout occurred after 3.1s
    5: timed out

3rd run (success)
➜  aws-sdk-rust-concurrency-issue git:(main) ✗ cargo r
    Finished dev [unoptimized + debuginfo] target(s) in 0.10s
     Running `target/debug/aws-sdk-rust-concurrent-issue`

4th run (dns error)
➜  aws-sdk-rust-concurrency-issue git:(main) ✗ cargo r
    Finished dev [unoptimized + debuginfo] target(s) in 0.09s
     Running `target/debug/aws-sdk-rust-concurrent-issue`
Error: Failed to collect objects

Caused by:
    0: Failed to fetch object 281
    1: dispatch failure
    2: io error
    3: error trying to connect: dns error: failed to lookup address information: nodename nor servname provided, or not known

This seems to indicate that we're pushing the use of underlying resources to the limit where it may or may not behave reliably. Could you explain what makes you use 200 (or more) concurrent GetObject?

from aws-sdk-rust.

Could you explain what makes you use 200 (or more) concurrent GetObject

Chiming in as well since this helps my current use case. We're trying to migrate millions of objects as fast as we can from a foreign S3 bucket (on a third party's AWS account), some of those objects being bigger than 800GB, so we'll be leveraging this blogpost to max out the amount of concurrent connections too, not only the amount of concurrent S3 Batch Operations objects.

OTOH, it seems that in order to implement the aforementioned AWS blogpost in Rust we'll be limited by this other issue first, though: #968, we'd need to have those parameters configurable:

max_concurrency: 940, max_retries: 100, max_pool_connections: 940 and multipart_chunksize: 16777216.

from aws-sdk-rust.

I found a resolution to the problem. I was actually expecting the SDK to return only two kinds of errors in this scenario: network issues with S3 itself or a failure to open a file descriptor due to a limit reached on my hardware.

While writing this response, I checked the maximum number of file descriptors I could open with the command ulimit -n, and it turns out to be 256 by default (at least on macOS). By removing this limit i.e. ulimit -n unlimited, I no longer encounter errors. 🙌

I'm closing this issue because the SDK behaves correctly. However it would be nice to add the root cause of this dns error for better clarity and troubleshooting in the future.

from aws-sdk-rust.

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

from aws-sdk-rust.