Comments (18)
Thanks for providing more details. Support for uploading unseekable streams in Transfer Utility
was added in S3 (3.7.202.0). You may want to upgrade to this version to confirm that it's an unseekable stream issue.
However, S3 (3.7.305.8) may have introduced some conflicts with the above feature release. Will discuss this with the team and keep you posted.
Regards,
Chaitanya
from aws-sdk-net.
Thank you for reporting the issue. Unfortunately, I am unable to reproduce the exception using your code sample.
Can you please elaborate on how the sourceStream
is populated? Preferably, if you can enable AWS SDK verbose logging that might also be helpful here. You can turn on verbose logging using the code below to see detailed logs.
Amazon.AWSConfigs.LoggingConfig.LogResponses = Amazon.ResponseLoggingOption.Always;
Amazon.AWSConfigs.LoggingConfig.LogTo = Amazon.LoggingOptions.SystemDiagnostics;
Amazon.AWSConfigs.AddTraceListener("Amazon", new System.Diagnostics.ConsoleTraceListener());
Regards,
Chaitanya
from aws-sdk-net.
Hello,
The sourceStream
is coming from a multipart form in an api request:
var boundary = Request.GetBoundary();
var reader = new MultipartReader(boundary, Request.Body, _bufferSize);
MultipartSection section;
while ((section = await reader.ReadNextSectionAsync(cancellationToken)) != null)
{
var contentDisposition = section.GetContentDispositionHeader();
if (contentDisposition.IsFileDisposition())
{
var fileSection = section.AsFileSection();
_logger.LogInformation("Received file to upload: {0}", fileSection.FileName);
Validate.FormFile(fileSection.FileStream, fileSection.FileName, fileSection.Section.ContentType);
var stream = new TypedStream
{
ContentType = fileSection.Section.ContentType,
ContentStream = fileSection.FileStream,
Filename = fileSection.FileName
};
await _documentManager.UploadDocument(tenantId, new DocumentReference(documentId), name, stream);
}
}
the _documentManager.UploadDocument
calls:
public async Task UploadDocument(Guid tenantId, IDocumentReference document,
string name, ITypedStream stream)
{
//... other code logic
await _storageManager.UploadContentStream(tenantId, document.DocumentId, name, stream);
}
the _storageManager.UploadContentStream
calls:
public async Task UploadContentStream(Guid tenantId, Guid id, string name, ITypedStream typedStream, CancellationToken cancellationToken = default)
{
var fullPath = $"{tenantId}/{id}/{name.ToLower()}";
await _blobStorageRepo.UploadAsync(typedStream.ContentStream, fullPath, typedStream.ContentType, cancellationToken);
_logger.LogInformation("Content stream has been uploaded");
}
the _blobStorageRepo.UploadAsync
is from a company's private nuget package that I do not have access to the code, I only can decompile and see what is called there (this is why I have the code snippet from the original question), so there is no possibility to turn on the detailed logging. Is there a way I can turn on this in this situation?
from aws-sdk-net.
One more thing. I altered the code
var boundary = Request.GetBoundary();
var reader = new MultipartReader(boundary, Request.Body, _bufferSize);
MultipartSection section;
while ((section = await reader.ReadNextSectionAsync(cancellationToken)) != null)
{
var contentDisposition = section.GetContentDispositionHeader();
if (contentDisposition.IsFileDisposition())
{
var fileSection = section.AsFileSection();
_logger.LogInformation("Received file to upload: {0}", fileSection.FileName);
Validate.FormFile(fileSection.FileStream, fileSection.FileName, fileSection.Section.ContentType);
var stream = new TypedStream
{
ContentType = fileSection.Section.ContentType,
ContentStream = fileSection.FileStream,
Filename = fileSection.FileName
};
await _documentManager.UploadDocument(tenantId, new DocumentReference(documentId), name, stream);
}
}
to this:
var fStream = System.IO.File.OpenRead(@"C:\original0001.pdf");
var stream = new TypedStream
{
ContentType = "application/pdf",
ContentStream = fStream,
Filename = "original0001.pdf"
};
await _documentManager.UploadDocument(tenantId, new DocumentReference(documentId), name, stream);
Basically I do not read the stream from the multipart form from the request but read a local file, and the Upload is successful. If this can help narrow the possible cause of the problem.
Regards,
from aws-sdk-net.
Hello,
I recently upgraded the nuget package to latest version and I have this same issue since.
I downgraded the package to 3.7.305.2 and that's fine.
I'm really interrested by this fix too.
The use case is same: upload from multipart form data without store anything into memory (case of big files like 10GB) and upload chunks to S3 directly without know the content length in advance.
All the best,
Adrien
from aws-sdk-net.
I'm also seeing the same issue.
from aws-sdk-net.
Looks like TransferUtility
doesn't support streams where CanSeek
is false. https://jasonterando.medium.com/net-core-tee-streaming-and-buffered-s3-uploads-4a063230d99f
from aws-sdk-net.
Related Issues (20)
- S3 Client not Exposing HeadBucket
- awssdk.s3 is missing NuGet package README file HOT 1
- awssdk.core is missing NuGet package README file HOT 1
- AWSSDK.Extensions.CrtIntegration AssemblyFileVersion is not updated HOT 1
- awssdk.extensions.netcore.setup is missing NuGet package README file HOT 1
- awssdk.dynamodbv2 is missing NuGet package README file HOT 1
- awssdk.sqs is missing NuGet package README file HOT 1
- awssdk.secretsmanager is missing NuGet package README file HOT 1
- awssdk.securitytoken is missing NuGet package README file HOT 3
- Cannot find a handler of type EndpointResolver HOT 17
- Missing functionality to craft "Conditions" in S3 Client when generating Pre-signed url HOT 2
- Getting Timeout error after updating from 3.7.300.29 to 3.7.301 and later versions HOT 8
- CopyObjectAsync throws AmazonUnmarshallingException when the SourceBucket is null HOT 3
- Proxy authentication using Kerberos HOT 2
- no_proxy environment variable is not respected HOT 6
- AmazonCloudWatchLogsClient.ListAnomaliesAsync throws Amazon.Runtime.AmazonClientException: We expected a VALUE token but got: ObjectStart HOT 14
- Get-S3Bucket : Invalid URI: The format of the URI could not be determined. HOT 15
- An option to accept email attachments as a CDN in the AWS's Simple Email Service HOT 3
- Set Content Type (mime code) for csv files HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aws-sdk-net.