Comments (15)
dgodwin1175
commented on September 13, 2024
1
Once the upload is cancelled, you can save the checkpoint like this:
try
{
return TransferManager.UploadAsync(file.FileName, destinationBlob, options, context, token);
}
catch (TaskCanceledException ex)
{
checkpoint = context.LastCheckpoint;
memoryStream = new MemoryStream();
formatter.Serialize(memoryStream, checkpoint);
file.Checkpoint = Convert.ToBase64String(memoryStream.ToArray());
throw ex;
}
And to resume your upload from the saved checkpoint:
memoryStream = new MemoryStream(Convert.FromBase64String(file.Checkpoint));
checkpoint = formatter.Deserialize(memoryStream) as TransferCheckpoint;
var context = new SingleTransferContext(checkpoint);
context.SetAttributesCallback = (destination) =>
{
var destBlob = destination as CloudBlob;
destBlob.Properties.ContentType = ContentType;
};
Then upload the file again with the above context object.
from azure-storage-net-data-movement.
johannesegger
commented on September 13, 2024
We added some debug output after saving the checkpoint, i.e. we used the Storage API to get all uncommitted blocks and matched it with recorder.
We get the following output:
Uncommitted blocks according to Storage API
A6689033-D96E-4F9B-B238-E27891C1B1FC/custom-upload-test3: 4194304 + 4194304 + 4194304 + 4194304 + 4194304 + 4194304 + 4194304 + 4194304 = 33554432 (32 MB)
Transfer statistics according to Data Movement Library
Transferred bytes: 33554432 (32 MB); Transfered: 0; Skipped: 0, Failed: 1
Uncommitted blocks according to Storage API
A6689033-D96E-4F9B-B238-E27891C1B1FC/custom-upload-test3: 4194304 + 4194304 + 4194304 + 4194304 + 4194304 + 4194304 + 4194304 + 4194304 + 4194304 + 4194304 + 4194304 = 46137344 (44 MB)
Transfer statistics according to Data Movement Library
Transferred bytes: 79691776 (76 MB); Transfered: 0; Skipped: 0, Failed: 2
Uncommitted blocks according to Storage API
A6689033-D96E-4F9B-B238-E27891C1B1FC/custom-upload-test3: 4194304 + 4194304 + 4194304 + 4194304 + 4194304 + 4194304 + 4194304 + 4194304 + 4194304 + 4194304 + 4194304 = 46137344 (44 MB)
Transfer statistics according to Data Movement Library
Transferred bytes: 121634816 (116 MB); Transfered: 0; Skipped: 0, Failed: 3
Uncommitted blocks according to Storage API
A6689033-D96E-4F9B-B238-E27891C1B1FC/custom-upload-test3: 4194304 + 4194304 + 4194304 + 4194304 + 4194304 + 4194304 + 4194304 + 4194304 + 4194304 + 4194304 + 4194304 = 46137344 (44 MB)
Transfer statistics according to Data Movement Library
Transferred bytes: 163577856 (156 MB); Transfered: 0; Skipped: 0, Failed: 4
To get the list of uncommitted blocks, we used the following snippet:
var blockList = blob.DownloadBlockList(BlockListingFilter.Uncommitted);
var expr = string.Join(" + ", blockList.Select(b => b.Length));
var size = blockList.Sum(b => b.Length);
Console.WriteLine($"{blob.Name}: {expr} = {size} ({size/1024.0/1024.0} MB)");
So it almost looks like we're not capturing the context correctly, isn't it?
from azure-storage-net-data-movement.
hilbert-tan
commented on September 13, 2024
Hi @johannesegger , did you end up finding the answer? I am encountering something similar to your problem, where the resume does not actually work. It restarts the whole transfer rather than continuing halfway.
from azure-storage-net-data-movement.
johannesegger
commented on September 13, 2024
I can't remember unfortunately, but I don't think so, sry.
from azure-storage-net-data-movement.
vinjiang
commented on September 13, 2024
@blueww , can you take a look?
from azure-storage-net-data-movement.
blueww
commented on September 13, 2024
@johannesegger
DMlib don't support resume when transfer is upload from stream , or download to stream. It only support resume upload/download when it's from/to a File.
If you use latest DMlib 0.7.1, you will see following exception when try to resume transfer from/to stream, and it indicate resume transfer from/to stream is not supported:
Microsoft.WindowsAzure.Storage.DataMovement.TransferException: Resuming transfer from or to a stream is not supported.
at Microsoft.WindowsAzure.Storage.DataMovement.TransferManager.GetTransfer(TransferLocation sourceLocation, TransferLocation destLocation, TransferMethod transferMethod, TransferContext transferContext) in C:\Local\Jenkins\jobs\DMLib_0.7.1\workspace\lib\TransferManager.cs:line 1329
at Microsoft.WindowsAzure.Storage.DataMovement.TransferManager.GetOrCreateSingleObjectTransfer(TransferLocation sourceLocation, TransferLocation destLocation, TransferMethod transferMethod, TransferContext transferContext) in C:\Local\Jenkins\jobs\DMLib_0.7.1\workspace\lib\TransferManager.cs:line 1275
at Microsoft.WindowsAzure.Storage.DataMovement.TransferManager.UploadAsync(Stream sourceStream, CloudBlob destBlob, UploadOptions options, SingleTransferContext context, CancellationToken cancellationToken) in C:\Local\Jenkins\jobs\DMLib_0.7.1\workspace\lib\TransferManager.cs:line 155
at ConsoleApp1.Program.Main(String[] args) in c:\users\weiwei\source\repos\ConsoleApp1\ConsoleApp1\Program.cs:line 48
from azure-storage-net-data-movement.
johannesegger
commented on September 13, 2024
That exception is very very helpful, thanks a lot.
However what's the problem with supporting resume from a stream? It should be possible if stream.CanSeek returns true, right?
from azure-storage-net-data-movement.
blueww
commented on September 13, 2024
In DMlib 0.7.1, we support none-seekable stream (upload to block blob), currently resume is not supported on all stream, no matter it's seekable or none seekable.
I have opened a task to track the feature request of supporting resume on seekable stream. We will evaluate the priority per the customer requests count on this feature.
from azure-storage-net-data-movement.
johannesegger
commented on September 13, 2024
Ok, thanks. I'm glad I can finally close this issue :-)
from azure-storage-net-data-movement.
thomaslevesque
commented on September 13, 2024
I have opened a task to track the feature request of supporting resume on seekable stream.
Hi @blueww, I can't find the issue you mentioned. Can you link to it?
from azure-storage-net-data-movement.
blueww
commented on September 13, 2024
@johannesegger
The task is a internal dev work track item for the DMlib. We will use the task to track all the dev work of DMlib.
So you might don't have access to it. Following is the link.
https://msazure.visualstudio.com/One/XStore-XClient/_workitems/edit/2997553
from azure-storage-net-data-movement.
johannesegger
commented on September 13, 2024
@thomaslevesque the above is for you
from azure-storage-net-data-movement.
thomaslevesque
commented on September 13, 2024
Ah, but it's not public, so the link is useless to me... @blueww any idea when this will be implemented?
from azure-storage-net-data-movement.
blueww
commented on September 13, 2024
We will priority this according to our bandwidth and other feature requests.
from azure-storage-net-data-movement.
haldarsumana
commented on September 13, 2024
Is there a way to save context.LastCheckpoint in the database or as a file to use it later to start blob upload from the last checkpoint by using Azure Data Movement library?
from azure-storage-net-data-movement.
Related Issues (20)
- Do not retry already failed files when restoring transfer from journal context
- Exception while creating Azure UserDelegation Key
- Send an Arbitrarily Large Stream to File Shares
- To download 267 MB files take 3 minutes
- Hang in DMLib when "System.Windows.Forms.Form" is loaded.
- Is there anything in the new API that replaces TransferManager.CopyDirectoryAsync ? HOT 1
- I am using this API for my Azure Data Lake Gen2 and it seems to work
- Method not found: 'Void Microsoft.Azure.Storage.Blob.BlobRequestOptions.set_EncryptionScope(System.String)
- Progress reporting reports different values for Upload/Download (a blob) and for download for a blob and a file share service HOT 1
- Trouble persisting to file a file or directory upload to file share
- A transfer hangs when two processes use Data Movement Library HOT 4
- How to write DownloadDirectoryAsync response in a Stream. HOT 1
- This repo is missing important files
- Copying files from one azure blob storage account to another
- Access to Azure File Share fails on Android (using .Net Maui)
- Support Status
- A file gets 409 error while data are committed HOT 1
- Include "Retry-After" in the Response Header when Graph Throttling occurs while moving the files from Sharepoint.
- BlobContainerClient.StartDownloadToDirectoryAsync url-encodes subdirectories containing spaces.
- Microsoft.Azure.Storage.DataMovement GPL Issue
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from azure-storage-net-data-movement.