S3Client instantiation extremely slow about aws-sdk-cpp HOT 8 OPEN

cc @fjetter

from aws-sdk-cpp.

Ok, so it seems the JSON parsing step that takes most of the time in the profile graphs may be spent parsing this JSON string hardcoded (!!) in the SDK's C++ source code:

It decodes into this heavily-nested object (here in Python representation):
https://gist.github.com/pitrou/c4978f29be9d2d3cb9574cd9d262490a

from aws-sdk-cpp.

Thanks for pointing this out. Currently it's behaving within expected boundaries, but I'm interested to hear more of your thoughts on this. How fast are you wanting/expecting the S3Client to instantiate? You shouldn't be needing to instantiate the that often as it can be reused.

I noticed in the issue linked above you have improved the performance of your tests to less than 53 µs with "caching PoC". Were there any changes you where wanting to be made on the sdk side?

from aws-sdk-cpp.

How fast are you wanting/expecting the S3Client to instantiate? You shouldn't be needing to instantiate the that often as it can be reused.

I'll let @fjetter elaborate on their situation, but when distributing individual tasks over a cluster of workers there's a need to deserialize everything that's needed to run such tasks. If a task entails loading data over S3 (with potentially different configurations, since tasks from multiple users or workloads might be in flight), it implies recreating a S3Client each time. Depending on task granularity, I suspect 1ms to instantiate a S3Client might appear as a significant contributor in performance profiles.

from aws-sdk-cpp.

For the record, here's the current prototype that seems to work on our CI. I ended up caching endpoint providers based on the S3 client configuration's relevant options (the ones that influence the provider initialization). There's an additional complication (InitOnceEndpointProvider) due to the fact that S3Client::S3Client always reconfigures the endpoint provider, even when it is explicitly passed by the caller, and that is not thread-safe.
https://github.com/apache/arrow/blob/2be51947448aac17da5eb4e7b284483da72f7f41/cpp/src/arrow/filesystem/s3fs.cc#L916-L1022

It would probably have been simpler if I could simply have explicitly created a shared RuleEngine with the S3 default rules, and instantiate each S3EndpointProvider from that same RuleEngine.

from aws-sdk-cpp.

How fast are you wanting/expecting the S3Client to instantiate? You shouldn't be needing to instantiate the that often as it can be reused.

I'll let @fjetter elaborate on their situation, but when distributing individual tasks over a cluster of workers there's a need to deserialize everything that's needed to run such tasks. If a task entails loading data over S3 (with potentially different configurations, since tasks from multiple users or workloads might be in flight), it implies recreating a S3Client each time. Depending on task granularity, I suspect 1ms to instantiate a S3Client might appear as a significant contributor in performance profiles.

That sums it up nicely. Due to how arrow and dask is built, we end up instantiating possibly thousands of clients adding up to a couple of seconds in latency whenever we're trying to read a dataset. We essentially end up creating one s3client per file, reusing it is a little difficult at this point.

from aws-sdk-cpp.

This is ultimately a feature request so I will be changing this issue to a feature-request. It's something that we would like to improve the speed of, but I don't have a timeline for when that might happen.

In the short term I would recommend to use a single endpoint resolver for all of you s3clients. You can do this by overloading the client when you initialize it.

from aws-sdk-cpp.

In the short term I would recommend to use a single endpoint resolver for all of you s3clients. You can do this by overloading the client when you initialize it.

We have a proposed workaround now, but it's slightly more complicated than that:

• we are a library, so cannot assume a single endpoint resolver since different s3clients may be requested for different endpoint configurations; we need to maintain a cache (which must also be appropriately flush betfore S3 shutdown)
• S3Client unfortunately does non-thread-safe reconfiguration of the given endpoint resolver in its constructor. That's harmless if the endpoint resolver is specific to a given client, not if the endpoint resolver is shared between multiple clients. We therefore had to implement an immutable endpoint resolver wrapper.

from aws-sdk-cpp.