Code Monkey home page Code Monkey logo

Comments (18)

bhoradc avatar bhoradc commented on July 18, 2024 2

Hi @julian-dimitroff,

Thanks for providing more details. Support for uploading unseekable streams in Transfer Utility was added in S3 (3.7.202.0). You may want to upgrade to this version to confirm that it's an unseekable stream issue.

However, S3 (3.7.305.8) may have introduced some conflicts with the above feature release. Will discuss this with the team and keep you posted.

Regards,
Chaitanya

from aws-sdk-net.

bhoradc avatar bhoradc commented on July 18, 2024

Hi @julian-dimitroff,

Thank you for reporting the issue. Unfortunately, I am unable to reproduce the exception using your code sample.

Can you please elaborate on how the sourceStream is populated? Preferably, if you can enable AWS SDK verbose logging that might also be helpful here. You can turn on verbose logging using the code below to see detailed logs.

Amazon.AWSConfigs.LoggingConfig.LogResponses = Amazon.ResponseLoggingOption.Always;
Amazon.AWSConfigs.LoggingConfig.LogTo = Amazon.LoggingOptions.SystemDiagnostics;
Amazon.AWSConfigs.AddTraceListener("Amazon", new System.Diagnostics.ConsoleTraceListener());

Regards,
Chaitanya

from aws-sdk-net.

julian-dimitroff avatar julian-dimitroff commented on July 18, 2024

Hello,
The sourceStream is coming from a multipart form in an api request:

var boundary = Request.GetBoundary();
var reader = new MultipartReader(boundary, Request.Body, _bufferSize);
MultipartSection section;
while ((section = await reader.ReadNextSectionAsync(cancellationToken)) != null)
{
    var contentDisposition = section.GetContentDispositionHeader();
    if (contentDisposition.IsFileDisposition())
    {
        var fileSection = section.AsFileSection();

        _logger.LogInformation("Received file to upload: {0}", fileSection.FileName);

        Validate.FormFile(fileSection.FileStream, fileSection.FileName, fileSection.Section.ContentType);
        var stream = new TypedStream
        {
            ContentType = fileSection.Section.ContentType,
            ContentStream = fileSection.FileStream,
            Filename = fileSection.FileName
        };
        await _documentManager.UploadDocument(tenantId, new DocumentReference(documentId), name, stream);
    }
}

the _documentManager.UploadDocument calls:

public async Task UploadDocument(Guid tenantId, IDocumentReference document,
    string name, ITypedStream stream)
{
    //... other code logic
    await _storageManager.UploadContentStream(tenantId, document.DocumentId, name, stream);
}

the _storageManager.UploadContentStream calls:

public async Task UploadContentStream(Guid tenantId, Guid id, string name, ITypedStream typedStream, CancellationToken cancellationToken = default)
{
	
	var fullPath = $"{tenantId}/{id}/{name.ToLower()}";
	await _blobStorageRepo.UploadAsync(typedStream.ContentStream, fullPath, typedStream.ContentType, cancellationToken);
	_logger.LogInformation("Content stream has been uploaded");
}

the _blobStorageRepo.UploadAsync is from a company's private nuget package that I do not have access to the code, I only can decompile and see what is called there (this is why I have the code snippet from the original question), so there is no possibility to turn on the detailed logging. Is there a way I can turn on this in this situation?

from aws-sdk-net.

julian-dimitroff avatar julian-dimitroff commented on July 18, 2024

One more thing. I altered the code

var boundary = Request.GetBoundary();
var reader = new MultipartReader(boundary, Request.Body, _bufferSize);
MultipartSection section;
while ((section = await reader.ReadNextSectionAsync(cancellationToken)) != null)
{
    var contentDisposition = section.GetContentDispositionHeader();
    if (contentDisposition.IsFileDisposition())
    {
        var fileSection = section.AsFileSection();

        _logger.LogInformation("Received file to upload: {0}", fileSection.FileName);

        Validate.FormFile(fileSection.FileStream, fileSection.FileName, fileSection.Section.ContentType);
        var stream = new TypedStream
        {
            ContentType = fileSection.Section.ContentType,
            ContentStream = fileSection.FileStream,
            Filename = fileSection.FileName
        };
        await _documentManager.UploadDocument(tenantId, new DocumentReference(documentId), name, stream);
    }
}

to this:

var fStream = System.IO.File.OpenRead(@"C:\original0001.pdf");
var stream = new TypedStream
{
    ContentType = "application/pdf",
    ContentStream = fStream,
    Filename = "original0001.pdf"
};
await _documentManager.UploadDocument(tenantId, new DocumentReference(documentId), name, stream);

Basically I do not read the stream from the multipart form from the request but read a local file, and the Upload is successful. If this can help narrow the possible cause of the problem.

Regards,

from aws-sdk-net.

acorbin-vdm avatar acorbin-vdm commented on July 18, 2024

Hello,
I recently upgraded the nuget package to latest version and I have this same issue since.
image

I downgraded the package to 3.7.305.2 and that's fine.
I'm really interrested by this fix too.

The use case is same: upload from multipart form data without store anything into memory (case of big files like 10GB) and upload chunks to S3 directly without know the content length in advance.

All the best,
Adrien

from aws-sdk-net.

laifharwood avatar laifharwood commented on July 18, 2024

I'm also seeing the same issue.

from aws-sdk-net.

laifharwood avatar laifharwood commented on July 18, 2024

Looks like TransferUtility doesn't support streams where CanSeek is false. https://jasonterando.medium.com/net-core-tee-streaming-and-buffered-s3-uploads-4a063230d99f

from aws-sdk-net.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.