S3Bucket does not support custom AwsClientParameters about prefect-aws HOT 6 CLOSED

I think we want to add here
https://github.com/PrefectHQ/prefect-aws/blob/main/prefect_aws/s3.py#L324-L341

Similar to:
https://github.com/PrefectHQ/prefect-aws/blob/main/prefect_aws/s3.py#L68-L69

from prefect-aws.

Actually, maybe it's not that straightforward since botocore.Config is not serializable as a pydantic field.

from prefect-aws.

No, serializing complex objects like botocore.Config or ssl.SSLContext is problematic - you could pickle and then use base64 to turn it all into a string, but how big is that string? I understand Blocks better now and my gut doesn't like this idea so much. Pydantic does support pickle, but to switch S3Bucket to using that is a bad idea.

But is it really necessary to serialize/store the Config object? How about this idea? It is a bit out-of-the-box, but it would work. Make S3Bucket.config a property and then there is Prefect's default implementation of the getter such that getting it actually creates it on the fly. Users could override this config getter with their own customizations. But their code would have to reassign the property every time it loads their special S3Bucket from the storage. Or the config property getter first looks for an optional create_custom() function attached to the object being manipulated. A small price to pay, but they are doing non-standard things.

Or just have S3Bucket.config be a function pointer such that internally, S3Bucket calls config() to get an actual Config. That can be overridden by user code. But that is basically the equiv of a property, just different syntax.

Pydantic is working on dealing with properties: pydantic/pydantic#935

If the creation function really needs to be stored, then the inspect module will return the actual source code and so it could be reconstructed upon loading from storage... and JSON will handle a text string. This is out of the box too, but I don't see any fundamental issues at the moment. S3Bucket would need a function to hook into Pydantic's json_encoder and a validator that understands how to reconstruct a string into a Config creation function, but that is not very complex.
For obsolete S3Buckets being read in, the validator could add the default create function etc. I am assuming any custom create function won't be very complex.

This can be solved, but what is the cost/benefit of adding this extra feature? I guess I'm the first to ask and others are just rolling their own solutions? I prefer my first idea.

from prefect-aws.

Would defining the attribute as

config: Optional[botocore.Config] = Field(None, exclude=True)

be acceptable? It gets documented and typed, but is prevented from being included in any export to JSON which it perhaps ought to be.

from prefect-aws.

If it's a Pydantic model/block, I think we want it to be portable; more info here https://orion-docs.prefect.io/ui/blocks/

from prefect-aws.

AwsCredentials now accepts client parameters.

from prefect-aws.