Can we have an async version of should overwrite callback? about azure-storage-net-data-movement HOT 9 CLOSED

@nrandell Which interface are you referencing to? Do you mean ShouldTransferCallback or ShouldOverwriteCallback?

from azure-storage-net-data-movement.

The ShouldOverwriteCallback - it's currently synchronous and if I want to do something such as calculate the MD5 of the file on disk, I have to block whilst doing the IO. I've got an async calculate MD5 and just want be able to await the result of that and then return true or false based on whether that is different or not.

from azure-storage-net-data-movement.

Are you worried about performance dropping by blocking transferring, or you'd want an asynchronous interface so that await is allowed in it?

DMLib would need result from ShouldOverwriteCallback to determine whether to continue the transferring that it cannot do anything else when waiting for ShouldOverwriteCallback for a single transfer item. If you are using DMLib to transfer multiple files, ShouldOverwriteCallback to each single transfer item is actually invoked parallely at the same time.

To invoke an async method in a synchronous callback, I think you can call task.Wait() or task.Result on your async method task according to its return value type.

from azure-storage-net-data-movement.

I'm looking for an async interface so await is allowed. Yes I can call task.Wait() or task.Result but that will end up blocking. My scenario involves transferring lots of new files each day. My assumption is that DMLib sets up a bunch of tasks to do the transfer and then the TPL kicks in doing all of it's nice asynchronous stuff to transfer the files.

In my mind the logic for each transfer would be something like. (I know the code isn't like this, but hopefully this explains my thinking)

var properties = await TryGetBlobPropertiesAsync();
if (!properties.Exists || !(await ShouldOverwriteCallbackAsync(blob))) {
  await DoTransferAsync();
}

This way there is no blocking at all, just a number of async calls, ultimately reducing the load on the machine.

from azure-storage-net-data-movement.

For multiple files, DMLib calls ShouldOverwriteCallback in multiple threads. DMLib also controls file count being transferred and transferring thread count. In your case, the ShouldOverwriteCallback for each individual file is called more or less at the same time. DMLib would want to block each individual transfer until ShouldOverwriteCallback returns, for it cannot proceed the transfer before got confirmation to overwrite existing destination.
You suggestion does make sense that to let TPL handle asynchronous stuff instead of doing it by DMLib itself.

from azure-storage-net-data-movement.

Please let us know if you need further assistance.

from azure-storage-net-data-movement.

So that would be a no then, despite @EmmaZhu saying that it makes sense to let TPL handle asynchronous stuff! Not impressed

from azure-storage-net-data-movement.

@nrandell Sorry for any misunderstanding.
ShouldOverwriteCallback is designed to block current file's transfer, without confirmation, it cannot proceed the following transfer work. But this callback only blocks the specified file, if there's other file transferring, it won't block them. Not sure whether this is what you'd want.

from azure-storage-net-data-movement.

I know it's designed to block the specified file, but what if your test needs to do something asynchronously, such as check the MD5 hash of something. You are going to have to do something like

bool ShouldOverwrite(object source, object destination) {
    var isOkToOverwrite = CheckIfItCanOverwriteAsync(source, destination).GetAwaiter().GetResult();
    return isOkToOverwrite;
}

This is fine if the calling is single threaded, but as DMLib is very asynchronous, this is really bad behaviour. See http://blog.stephencleary.com/2012/07/dont-block-on-async-code.html for more details.

When I created the issue, I only wanted to overwrite files that were different. In order to do that, I needed to check the MD5 of the file on disk. This is done asynchronously and I just wanted to call it cleanly using all the standard async goodies that the majority of .Net libraries has started moving to.

Using GetAwaiter().GetResult() basically blocks until the task has finished, effectively using a thread which is expensive for the CPU and scalability.

from azure-storage-net-data-movement.