Setting CopyOptions for Incremental Copy about azure-storage-net-data-movement HOT 3 CLOSED

Hi, Mark

I suppose you are using CopyAsync(CloudBlob sourceBlob, CloudBlob destBlob, bool isServiceCopy, CopyOptions options, TransferContext context). Since you already got sourceBlob & destBlob before calling CopyAsync, you can fetch the attribute of them and decide whether to do the copy or not according to their LastModified properties.

By default, DML will not overwrite an existing destination, you can force the overwrite by provide a OverwriteCallback in TransferContext. Here's the code snippet for your reference:

        // Fetch blob attributes from server
        sourceBlob.FetchAttributes();
        destBlob.FetchAttributes();

        // Check if the source blob is newer than the destination blob.
        if (sourceBlob.Properties.LastModified > destBlob.Properties.LastModified)
        {
            TransferContext transferContext = new TransferContext()
            {
                OverwriteCallback = (source, destination) =>
                {
                    // This callback is triggered when DML find the destination already exists.
                    // Return true to overwrite the destination, false to skip.
                    return true;
                }
            };

            await TransferManager.CopyAsync(sourceBlob, destBlob, false /* isServiceCopy */, null /* options */, transferContext);
        }

from azure-storage-net-data-movement.

Interesting. Thanks for this.

So when doing an incremental /XO type of copy then do I even need to use or set CopyOptions?

It seems to me I could just set the Overwrite delegate to a function and then in my delegate check whether I'm doing an incremental copy in the first place and if I am, then check if the source is newer than the destination by checking the blob properties using the CloudBlob objects hydrated from the URI's passed into the delegate.

Is that correct?

Thanks,
Mark

from azure-storage-net-data-movement.

Correct. CopyOptions provide a way to set conditional header in requests sent to source/destination blobs and it doesn't help in this scenario.

You can either check whether the source is newer than destination before call DML or inside the Overwrite delegate as you mentioned. A small difference is if you do it inside the Overwrite delegate and returns false to not overwrite the destination. The corresponding Task return by CopyAsync will end up with an exception indicating the copy is skipped.

Best Regards,
Longwen

from azure-storage-net-data-movement.