frendsplatform / frends.amazons3 Goto Github PK

Frends doesn't notice anything wrong with uploading files larger than 5GB: the processes run just fine, the files just simply don't show up in the S3 bucket. We found articles (e.g. https://saturncloud.io/blog/how-to-upload-large-files-to-amazon-s3-as-a-data-scientist/) describing ways of using an AWS SDK for chopping the upload into multiple smaller pieces.
This is possible to implement with Multipart uploading: https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html

Currently library does not support multipart upload, but it might not activated settings for this. It should be investigated how big process is to update current task or whether new task should be created with multipart upload.

https://github.com/CommunityHiQ/Frends.Community.AWS

Need to create test AWS creds

PDF files are downloaded without exception but they are blank.

https://github.com/CommunityHiQ/Frends.Community.AWS

Need to create test AWS creds

A request has been implemented to Frends.Community.AWS.UploadFiles in which the Task will provide debug log of the file transfer as result of the Task: CommunityHiQ/Frends.Community.AWS#22.

This feature helps debugging issues with the file transfer which are not necessarily issues with the Task, but for example AWS S3 configurations or some other things related to S3.

Implementation proposal: CommunityHiQ/Frends.Community.AWS@cbfc6f8.

We have upload, download and list, but no delete yet.

https://github.com/CommunityHiQ/Frends.Community.AWS

Need to create test AWS creds

RnD report says that it is not unloaded, and this causes problems with agent memory. Related FrendsPlatform/FrendsTasks#32