[VI]: after initial migration, some assets will be marked as failed. it would be good to have an option to rerun only failed jobs after fixes.

Customer can choose their preferred manifest name for .mpd and .m3u8 for the generated asset, if it is not set, use the input asset's manifest name.

For commands that deal with AMS assets, below two command line options are common and useful to select a list of input assets without filter option with the special format.

--creation-time-start (-cs) is the earliest creation time of the selected input assets.
--creation-time-end (-ce) is the latest creation time of the selected input assets.

Make the -resource-filter to cover less common scenarios.

Storage container name is limited to 64 characters but asset names can be longer than that. Also storage container names are lower case only while asset names are mixed case. Lower case and truncate to 64 characters and report the truncation of container name.

Shaka packager fails to parse non-interleaved fragmented MP4. Right now we have a fix for shaka packager in works but if that takes longer we might want to run ffmpeg to transmux the content into a form that is easy to handle by shaka packager.

I hit this issue while trying to migrate nimbuspm, and wanted to raise it here (I’ll also put it in github) as a question to see what we should do?

Running this:

./AMSMigrate assets -s 2b461b25-f7b4-4a22-90cc-d640a14b5471 -g johndeu_DO_NOT_DELETE -n nimbuspm -o https://pohhsustoragetest.blob.core.windows.net/ -t test/${AssetName}-stream-dash --skip-migrated False --copy-nonstreamable False --working-dir /datadrive/AMSMigrate

AMSMigrate hit this exception and exited:

[05:26:02 Info] Migrating asset: 02479cd2-82cc-4d46-8276-1c8cbddd1bef (container asset-02479cd2-82cc-4d46-8276-1c8cbddd1bef) ...

Asset Migration ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 630/2159 Assets 29% 00:04:59 ⣯
Unhandled exception: Azure.RequestFailedException: The specified container does not exist.
RequestId:8f874d35-201e-0021-4793-b0a279000000
Time:2023-07-07T05:26:02.3324457Z
Status: 404 (The specified container does not exist.)
ErrorCode: ContainerNotFound

Content:

RequestId:8f874d35-201e-0021-4793-b0a279000000
Time:2023-07-07T05:26:02.3324457Z

Headers:
Vary: Origin
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
x-ms-request-id: 8f874d35-201e-0021-4793-b0a279000000
x-ms-client-request-id: deec121d-dc41-45a3-b6c5-b4cfdc252692
x-ms-version: 2022-11-02
x-ms-error-code: ContainerNotFound
Date: Fri, 07 Jul 2023 05:26:01 GMT
Content-Length: 225
Content-Type: application/xml

at Azure.Storage.Blobs.ContainerRestClient.GetPropertiesAsync(Nullable1 timeout, String leaseId, CancellationToken cancellationToken) at Azure.Storage.Blobs.BlobContainerClient.GetPropertiesInternal(BlobRequestConditions conditions, Boolean async, CancellationToken cancellationToken) at Azure.Storage.Blobs.BlobContainerClient.GetPropertiesAsync(BlobRequestConditions conditions, CancellationToken cancellationToken) at AMSMigrate.Ams.AssetMigrationTracker.GetMigrationStatusAsync(BlobContainerClient container, CancellationToken cancellationToken) in /home/pohhsu/z/azure-media-migration/ams/AssetMigrationTracker.cs:line 68 at AMSMigrate.Ams.AssetMigrator.MigrateAsync(MediaServicesAccountResource account, BlobServiceClient storage, MediaAssetResource asset, CancellationToken cancellationToken) in /home/pohhsu/z/azure-media-migration/ams/AssetMigrator.cs:line 144 at AMSMigrate.Ams.AssetMigrator.<>c__DisplayClass6_0.<<MigrateAsync>b__1>d.MoveNext() in /home/pohhsu/z/azure-media-migration/ams/AssetMigrator.cs:line 89 --- End of stack trace from previous location --- at AMSMigrate.Ams.AssetMigrator.<>c__DisplayClass6_0.<<MigrateAsync>b__0>d.MoveNext() in /home/pohhsu/z/azure-media-migration/ams/AssetMigrator.cs:line 89 --- End of stack trace from previous location --- at AMSMigrate.Ams.BaseMigrator.MigrateInBatches[T](AsyncPageable1 pageable, List1 filteredList, Func2 processBatch, Int32 batchSize, CancellationToken cancellationToken) in /home/pohhsu/z/azure-media-migration/ams/BaseMigrator.cs:line 61
at AMSMigrate.Ams.BaseMigrator.MigrateInBatches[T](AsyncPageable1 pageable, List1 filteredList, Func2 processBatch, Int32 batchSize, CancellationToken cancellationToken) in /home/pohhsu/z/azure-media-migration/ams/BaseMigrator.cs:line 57 at AMSMigrate.Ams.AssetMigrator.MigrateAsync(MediaServicesAccountResource account, AsyncPageable1 assets, List1 filteredList, ChannelWriter1 writer, CancellationToken cancellationToken) in /home/pohhsu/z/azure-media-migration/ams/AssetMigrator.cs:line 87
at AMSMigrate.Ams.AssetMigrator.MigrateAsync(CancellationToken cancellationToken) in /home/pohhsu/z/azure-media-migration/ams/AssetMigrator.cs:line 76
at AMSMigrate.Program.MigrateAssetsAsync(GlobalOptions globalOptions, AssetOptions assetOptions, CancellationToken cancellationToken) in /home/pohhsu/z/azure-media-migration/Program.cs:line 186
at AMSMigrate.Program.<>c__DisplayClass0_0.<

The Asset name is this: 02479cd2-82cc-4d46-8276-1c8cbddd1bef

It turns out the container (asset-02479cd2-82cc-4d46-8276-1c8cbddd1bef) is non-existing…

So wondering if this is okay to exit as is? Or we need to add code to catch this exception and then just fail and continue? (also will raise issue in github)

This is to follow the document review comments for command line options.

  Rename the global --account-name with --source-account-name,  and make it per-command option,
  the description for this option is different for each command.

  Rename --storage-account option with --output-storage-account for "assets" and "storage" commands.

  Move the global option --resource-filter to per-command option, it is used for the AMS assets only.

  The options for "assets" and "storage" are not exact same, so introduce new set of classes for "storag" command.

  Add a --prefix for "storage" command.

  Make option --packager for "assets" and "storage" as hidden for now, the customer cannot choose a packager in current version.

  More command specific options will be added later.

Sure. Approved the PR.
-Prakash

From: Pohsiang (John) Hsu <[email protected]>
Sent: Monday, June 26, 2023 11:36 AM
To: Prakash Duggaraju <[email protected]>
Cc: Bruce Lin <[email protected]>; WeiBing Zhan <[email protected]>; Melinda Wang <[email protected]>; Anirban Roy <[email protected]>; Yubo Xie <[email protected]>
Subject: RE: AMSMigrate fail to upload when top level container is not created before hand?

@Prakash,

Not sure if you have decided on the optimal code path yet…
So to unblock testing, I suggest we can temporarily do the naïve approach of check for container existence or not… please let me know if this is okay or not?
check if output container asset exists or not, if not, create it by pohhsu · Pull Request #15 · Azure/azure-media-migration (github.com)

From: Prakash Duggaraju <[email protected]>
Sent: Thursday, June 22, 2023 12:16 PM
To: Pohsiang (John) Hsu <[email protected]>
Cc: Bruce Lin <[email protected]>; WeiBing Zhan <[email protected]>; Melinda Wang <[email protected]>; Anirban Roy <[email protected]>; Yubo Xie <[email protected]>
Subject: RE: AMSMigrate fail to upload when top level container is not created before hand?

Right now it is missing one line in code.
Adding await container.CreateIfNotExistsAsync() to AzureStorageUplaoder.cs would fix this.
But I wasn’t sure if we want to do this for every single file or if we want to optimize it by remembering the container created so if uploading to same container, we only need to call this once.

-Prakash.

From: Pohsiang (John) Hsu <[email protected]>
Sent: Thursday, June 22, 2023 11:12 AM
To: Prakash Duggaraju <[email protected]>
Cc: Bruce Lin <[email protected]>; WeiBing Zhan <[email protected]>; Melinda Wang <[email protected]>; Anirban Roy <[email protected]>; Yubo Xie <[email protected]>
Subject: AMSMigrate fail to upload when top level container is not created before hand?

@Prakash,

Running this command (with the highlighted pattern)

AMSMigrate.exe assets -s b2c12710-ecdf-4532-9ba6-7e74a219b5f2 -g pohhsuTest -n pohhsumediaservice -o pohhsumediaservice -t test-output-1/${AssetName}-stream-dash -f "name eq '0365video-03f3f58678-StandardEncoder-H265ContentAwareEncoding-042d34d192'"

Fails with this:

[11:05:23 AM Fail] Failed to migrate asset 0365video-03f3f58678-StandardEncoder-H265ContentAwareEncoding-042d34d192 Error:Azure.RequestFailedException: The specified container does not exist.
RequestId:038ac1db-401e-000d-4434-a564e9000000━━━━━━━━━━ 0/40 Assets 0% 00:00:55 ⣟
Time:2023-06-22T18:04:35.7213969Z
Status: 404 (The specified container does not exist.)
ErrorCode: ContainerNotFound

Content:

RequestId:038ac1db-401e-000d-4434-a564e9000000
Time:2023-06-22T18:04:35.7213969Z

Headers:
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
x-ms-request-id: 038ac1db-401e-000d-4434-a564e9000000
x-ms-client-request-id: b9006f32-e432-494b-9ada-14267d810579
x-ms-version: 2022-11-02
x-ms-error-code: ContainerNotFound
Date: Thu, 22 Jun 2023 18:04:34 GMT
Content-Length: 225
Content-Type: application/xml

at Azure.Storage.Blobs.BlobRestClient.StartCopyFromURLAsync(String copySource, Nullable1 timeout, IDictionary2 metadata, Nullable1 tier, Nullable1 rehydratePriority, Nullable1 sourceIfModifiedSince, Nullable1
sourceIfUnmodifiedSince, String sourceIfMatch, String sourceIfNoneMatch, String sourceIfTags, Nullable1 ifModifiedSince, Nullable1 ifUnmodifiedSince, String ifMatch, String ifNoneMatch, String ifTags, String leaseId, String
blobTagsString, Nullable1 sealBlob, Nullable1 immutabilityPolicyExpiry, Nullable1 immutabilityPolicyMode, Nullable1 legalHold, CancellationToken cancellationToken)
at Azure.Storage.Blobs.Specialized.BlobBaseClient.StartCopyFromUriInternal(Uri source, IDictionary2 metadata, IDictionary2 tags, Nullable1 accessTier, BlobRequestConditions sourceConditions, BlobRequestConditions destinationConditions, Nullable1 rehydratePriority, Nullable1 sealBlob, BlobImmutabilityPolicy destinationImmutabilityPolicy, Nullable1 legalHold, Boolean async, CancellationToken cancellationToken)
at Azure.Storage.Blobs.Specialized.BlobBaseClient.StartCopyFromUriAsync(Uri source, IDictionary2 metadata, Nullable1 accessTier, BlobRequestConditions sourceConditions, BlobRequestConditions destinationConditions, Nullable1 rehydratePriority, CancellationToken cancellationToken) at AMSMigrate.Azure.AzureStorageUploader.UploadBlobAsync(String containerName, String fileName, BlockBlobClient blob, CancellationToken cancellationToken) in D:\z\azure-media-migration\azure\AzureStorageUploader.cs:line 67 at AMSMigrate.Transform.StorageTransform.UploadBlobAsync(BlockBlobClient blob, ValueTuple2 outputPath, CancellationToken cancellationToken) in D:\z\azure-media-migration\transform\StorageTransform.cs:line 85
at AMSMigrate.Transform.PackageTransform.<>c__DisplayClass3_0.<b__1>d.MoveNext() in D:\z\azure-media-migration\transform\PackageTransform.cs:line 60
--- End of stack trace from previous location ---
at AMSMigrate.Transform.PackageTransform.TransformAsync(AssetDetails details, ValueTuple`2 outputPath, CancellationToken cancellationToken) in D:\z\azure-media-migration\transform\PackageTransform.cs:line 128

What I noticed is that if I go and create a container called test-ouput-1 in pohhsumediaservice first, then it will succeeds.

Question is if this by design for this path-template , i.e. the user needs to create the top level container (i.e. ‘test-output-1’ in this example) in storage account first? Or did we just miss creation of the top level container?

Run following command, only one asset is converted & failed to migrate asset asset-299c688316
asset-299c688316 plays fine on portal player but AMSMigrate.exe failed to migrate

AMSMigrate.exe assets -s 2b461b25-f7b4-4a22-90cc-d640a14b5471 -g johndeu_DO_NOT_DELETE -n nimbuspm -o pohhsumediaservice -t blinmay-output/${AssetName}-stream-dash -cs 2022-04-30 -ce 2022-05-30

Expecting at least 2 out of 3 assets (playing fine on portal player) should be converted. but only 1 is converted successfully.

Failed to migrate asset asset-299c688316

5:11:25 PM Fail] Failed to migrate asset asset-299c688316 Error:System.IO.IOException: Pipe is broken.
at System.IO.Pipes.PipeStream.PipeValueTaskSource.GetResult(Int16 token):04:38 ⢿
at System.IO.Pipes.PipeStream.PipeValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
at System.IO.Stream.g__Core|29_0(Stream source, Stream destination, Int32 bufferSize, CancellationToken cancellationToken)
at Azure.Storage.Blobs.PartitionedDownloader.CopyToAsync(BlobDownloadStreamingResult result, Stream destination, CancellationToken cancellationToken)
at Azure.Storage.Blobs.PartitionedDownloader.<>c__DisplayClass9_0.<g__ConsumeQueuedTask|0>d.MoveNext()
--- End of stack trace from previous location ---
at Azure.Storage.Blobs.PartitionedDownloader.DownloadToAsync(Stream destination, BlobRequestConditions conditions, CancellationToken cancellationToken)
at Azure.Storage.Blobs.Specialized.BlobBaseClient.StagedDownloadAsync(Stream destination, BlobRequestConditions conditions, IProgress1 progressHandler, StorageTransferOptions transferOptions, DownloadTransferValidationOptions transferValidationOverride, Boolean async, CancellationToken cancellationToken) at Azure.Storage.Blobs.Specialized.BlobBaseClient.DownloadToAsync(Stream destination, BlobRequestConditions conditions, StorageTransferOptions transferOptions, CancellationToken cancellationToken) at Azure.Storage.Blobs.Specialized.BlobBaseClient.DownloadToAsync(Stream destination, CancellationToken cancellationToken) at AMSMigrate.Pipes.BlobStream.DownloadAsync(Stream stream, CancellationToken cancellationToken) in D:\Packager\azure-media-migration\pipes\BlobPipe.cs:line 27 at AMSMigrate.Pipes.BlobPipe.<DownloadAsync>b__2_0(Stream stream, CancellationToken token) in D:\Packager\azure-media-migration\pipes\BlobPipe.cs:line 57 at AMSMigrate.Pipes.Pipe.RunAsync(Func3 action, CancellationToken cancellationToken) in D:\Packager\azure-media-migration\pipes\Pipe.cs:line 88
at AMSMigrate.Pipes.BlobPipe.DownloadAsync(CancellationToken cancellationToken) in D:\Packager\azure-media-migration\pipes\BlobPipe.cs:line 55
at AMSMigrate.Pipes.BlobPipe.RunAsync(CancellationToken cancellationToken) in D:\Packager\azure-media-migration\pipes\BlobPipe.cs:line 71
at AMSMigrate.Transform.PackageTransform.<>c__DisplayClass3_0.<b__6>d.MoveNext() in D:\Packager\azure-media-migration\transform\PackageTransform.cs:line 75
--- End of stack trace from previous location ---
at AMSMigrate.Transform.PackageTransform.TransformAsync(AssetDetails details, ValueTuple`2 outputPath, CancellationToken cancellationToken) in D:\Packager\azure-media-migration\transform\PackageTransform.cs:line 134

Right now the tool assumes a 1:1 correlation between AMS account and storage account. If a single AMS account is used with multiple storage accounts then the tool needs to be enhanced to work correctly.

This repository is currently missing a LICENSE.MD file outlining its license. A license helps users understand how to use your project in a compliant manner. You can find the standard MIT license text at the Microsoft repo templates LICENSE file: https://github.com/microsoft/repo-templates/blob/main/shared/LICENSE.
If you would like to learn more about open source licenses, please visit the document at https://aka.ms/license.

on WSL, build and then run the following command with illegal asset name (Test-output)

pohhsu@pohhsu3:~/z/azure-media-migration/bin/Debug/net6.0$ chmod +x ./packager-linux-x64
pohhsu@pohhsu3:~/z/azure-media-migration/bin/Debug/net6.0$ ./AMSMigrate assets -s b2c12710-ecdf-4532-9ba6-7e74a219b5f2 -g pohhsuTest -n pohhsumediaservice -o pohhsumediaservice -t Test-output/${AssetName}-stream-dash -f "name eq '0365video-03f3f58678-StandardEncoder-H265ContentAwareEncoding-042d34d192'"

the program hangs

[19:55:36 Info] Begin migration of assets for account: pohhsumediaservice
[19:55:38 Info] Migrating asset: 0365video-03f3f58678-StandardEncoder-H265ContentAwareEncoding-042d34d192 ...
[19:55:40 Info] [0627/195540:INFO:demuxer.cc(89)] Demuxer::Run() on file
'/tmp/AMSMigrate/0365video-03f3f58678-StandardEncoder-H265ContentAwareEncoding-042d34d192/O365Video_Azure1080p_1280x720_AACAudio_246.mp4'.
[19:55:40 Info] [0627/195540:INFO:demuxer.cc(155)] Initialize Demuxer for file
'/tmp/AMSMigrate/0365video-03f3f58678-StandardEncoder-H265ContentAwareEncoding-042d34d192/O365Video_Azure1080p_1280x720_AACAudio_246.mp4'.
[19:55:41 Fail] Packager {} finished with exit code 2

This is on Azure Linux VM: @duggaraju , any ideas?

[22:02:35 Dbug] Starting packager /home/pohhsu/test/publish/packager-linux-x64...

…
Unhandled exception. System.AggregateException: An error occurred while writing to logger(s). (Object reference not set to an instance of an object.)
---> System.NullReferenceException: Object reference not set to an instance of an object.
at Vertical.SpectreLogger.Templates.TemplateSegment.get_CompositeFormatSpan()
at Vertical.SpectreLogger.Output.WriteBufferExtensions.WriteLogValue[T](IWriteBuffer buffer, LogLevelProfile profile, TemplateSegment templateSegment, T value, Action1 writer) at Vertical.SpectreLogger.Rendering.RendererPipeline.Render(LogEventContext& logEventContext) at Vertical.SpectreLogger.SpectreLogger.Log[TState](LogLevel logLevel, EventId eventId, TState state, Exception exception, Func3 formatter)
at Microsoft.Extensions.Logging.Logger.g__LoggerLog|13_0[TState](LogLevel logLevel, EventId eventId, ILogger logger, Exception exception, Func3 formatter, List1& exceptions, TState& state)
--- End of inner exception stack trace ---
at Microsoft.Extensions.Logging.Logger.ThrowLoggingError(List1 exceptions) at Microsoft.Extensions.Logging.Logger.Log[TState](LogLevel logLevel, EventId eventId, TState state, Exception exception, Func3 formatter)
at Microsoft.Extensions.Logging.Logger1.Microsoft.Extensions.Logging.ILogger.Log[TState](LogLevel logLevel, EventId eventId, TState state, Exception exception, Func3 formatter)
at Microsoft.Extensions.Logging.LoggerExtensions.Log(ILogger logger, LogLevel logLevel, EventId eventId, Exception exception, String message, Object[] args)
at Microsoft.Extensions.Logging.LoggerExtensions.Log(ILogger logger, LogLevel logLevel, String message, Object[] args)
at AMSMigrate.Transform.ShakaPackager.LogStandardError(String line) in /home/pohhsu/z/azure-media-migration/transform/ShakaPackager.cs:line 119
at AMSMigrate.Transform.BasePackager.<>c__DisplayClass44_0.b__1(Object s, DataReceivedEventArgs args) in /home/pohhsu/z/azure-media-migration/transform/BasePackager.cs:line 123
at System.Diagnostics.AsyncStreamReader.FlushMessageQueue(Boolean rethrowInNewThread)
--- End of stack trace from previous location ---
at System.Diagnostics.AsyncStreamReader.<>c.b__18_0(Object edi)
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
../run.sh: line 3: 3366 Aborted (core dumped) ./AMSMigrate assets -s 2b461b25-f7b4-4a22-90cc-d640a14b5471 -g johndeu_DO_NOT_DELETE -n nimbuspm -o https://pohhsustoragetest.blob.core.windows.net -t test7/${AssetName}-stream-dash --skip-migrated False --copy-nonstreamable False --working-dir /datadrive/AMSMigrate

Ability to statically encrypt the content while packaging and save the key to a key vault with the key ID as the secret name. The key should be part of the license URL template that is passed as command line argument.

dependent on #107

@Prakash, I’ve hit this sporadically (not 100% repro but repro rate is high enough). Wondering if you have some ideas about this exception…

I’m running Analyze on nimbuspm without the Storage Blob Data Contributor role to the storage account. So a lot of expected exceptions with no authorization.

command line> AMSMigrate.exe analyze -s 2b461b25-f7b4-4a22-90cc-d640a14b5471 -g johndeu_DO_NOT_DELETE -n nimbuspm

When it works, it will finish analyzing, but every once in a while, it will fail in

       Unhandled exception: System.AggregateException: An error occurred while writing to logger(s). (An item with the same key has already been added. Key: Vertical.SpectreLogger.Rendering.ExceptionRenderer+Options)

---> System.ArgumentException: An item with the same key has already been added. Key: Vertical.SpectreLogger.Rendering.ExceptionRenderer+Options

Full console output below: (I’ll enter an issue shortly)

AMSMigrate.exe analyze -s 2b461b25-f7b4-4a22-90cc-d640a14b5471 -g johndeu_DO_NOT_DELETE -n nimbuspm
[1:06:21 PM Dbug] Writing logs to D:\z\azure-media-migration\bin\Debug\net6.0\MigrationLog_13_06_21.txt
[1:06:21 PM Info] Begin analysis of assets for account: nimbuspm
[1:06:39 PM Info] The total asset count of the media account is 2159.
[1:06:39 PM Info] The total assets to handle in this run is 2159.
[1:06:40 PM Dbug] Analyzing asset: 00175c82-7580-49e9-bfe9-b0de7e93799c (container asset-00175c82-7580-49e9-bfe9-b0de7e93799c)
[1:06:40 PM Dbug] Analyzing asset: 005955fd-1825-47c3-bf12-1a55d059aaf0 (container asset-53647991-76e0-40da-91ba-eee394641b74)
[1:06:40 PM Dbug] Analyzing asset: 00611e75-2998-4f62-a5b5-6228ba5df31a (container asset-00611e75-2998-4f62-a5b5-6228ba5df31a)
[1:06:40 PM Dbug] Analyzing asset: 0094bc8d-fc8a-441d-9eb5-7166fcc061dd (container asset-b20b4d60-7b45-4eb8-bc13-4e21b33f561f)
[1:06:40 PM Dbug] Analyzing asset: 0101ab57-51f4-4f69-9c38-252c46d1d1e9 (container asset-0101ab57-51f4-4f69-9c38-252c46d1d1e9)
Markup error: Unbalanced markup stack. Did you forget to close a tag?
Tried to write:
[grey85][[[grey85]1:06:46 PM[/] [red1]Fail[/]]] Failed to analyze asset [gold3_1]005955fd-1825-47c3-bf12-1a55d059aaf0[/]
Analyzing Assets ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/2159 Assets   0% 00:00:06 ⢿
[1:06:46 PM Fail] Failed to analyze asset 0101ab57-51f4-4f69-9c38-252c46d1d1e9
Azure.RequestFailedException: This request is not authorized to perform this operation using this permission.
RequestId:12209436-101e-0048-55c5-aa9b35000000
Time:2023-06-29T20:06:46.3826947Z
Status: 403 (This request is not authorized to perform this operation using this permission.)
ErrorCode: AuthorizationPermissionMismatch

Content:
<?xml version="1.0" encoding="utf-8"?><Error><Code>AuthorizationPermissionMismatch</Code><Message>This request is not authorized to perform this operation using this permission.
RequestId:12209436-101e-0048-55c5-aa9b35000000
Time:2023-06-29T20:06:46.3826947Z</Message></Error>

Headers:
Vary: Origin
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
x-ms-request-id: 12209436-101e-0048-55c5-aa9b35000000
x-ms-client-request-id: ea7f57d7-4af2-4e1b-ba43-3f29674b2c8a
x-ms-version: 2022-11-02
x-ms-error-code: AuthorizationPermissionMismatch
Date: Thu, 29 Jun 2023 20:06:46 GMT
Content-Length: 279
Content-Type: application/xml

   at Azure.Storage.Blobs.<ListBlobHierarchySegmentAsync>d__59.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaiter.GetResult()
   +29 more...
[1:06:46 PM Fail] Failed to analyze asset 0094bc8d-fc8a-441d-9eb5-7166fcc061dd
Azure.RequestFailedException: This request is not authorized to perform this operation using this permission.
RequestId:aec62b17-201e-007c-57c5-aaa8fd000000
Time:2023-06-29T20:06:46.4111744Z
Status: 403 (This request is not authorized to perform this operation using this permission.)
ErrorCode: AuthorizationPermissionMismatch

Content:
<?xml version="1.0" encoding="utf-8"?><Error><Code>AuthorizationPermissionMismatch</Code><Message>This request is not authorized to perform this operation using this permission.
RequestId:aec62b17-201e-007c-57c5-aaa8fd000000
Time:2023-06-29T20:06:46.4111744Z</Message></Error>

Headers:
Vary: Origin
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
x-ms-request-id: aec62b17-201e-007c-57c5-aaa8fd000000
x-ms-client-request-id: 943e4c95-32ac-4cd3-af17-8d3cd3a8a760
x-ms-version: 2022-11-02
x-ms-error-code: AuthorizationPermissionMismatch
Date: Thu, 29 Jun 2023 20:06:46 GMT
Content-Length: 279
Content-Type: application/xml

   at Azure.Storage.Blobs.<ListBlobHierarchySegmentAsync>d__59.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaiter.GetResult()
   +29 more...
[1:06:46 PM Fail] Failed to analyze asset 00175c82-7580-49e9-bfe9-b0de7e93799c
Azure.RequestFailedException: This request is not authorized to perform this operation using this permission.
RequestId:ffe36224-d01e-000a-26c5-aa22b5000000
Time:2023-06-29T20:06:46.5075729Z
Status: 403 (This request is not authorized to perform this operation using this permission.)
ErrorCode: AuthorizationPermissionMismatch

Content:
<?xml version="1.0" encoding="utf-8"?><Error><Code>AuthorizationPermissionMismatch</Code><Message>This request is not authorized to perform this operation using this permission.
RequestId:ffe36224-d01e-000a-26c5-aa22b5000000
Time:2023-06-29T20:06:46.5075729Z</Message></Error>

Headers:
Vary: Origin
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
x-ms-request-id: ffe36224-d01e-000a-26c5-aa22b5000000
x-ms-client-request-id: fe28a9a3-9c0f-46e4-9e66-5646711c8140
x-ms-version: 2022-11-02
x-ms-error-code: AuthorizationPermissionMismatch
Date: Thu, 29 Jun 2023 20:06:46 GMT
Content-Length: 279
Content-Type: application/xml

   at Azure.Storage.Blobs.<ListBlobHierarchySegmentAsync>d__59.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaiter.GetResult()
   +29 more...
[1:06:46 PM Fail] Failed to analyze asset 00611e75-2998-4f62-a5b5-6228ba5df31a
Azure.RequestFailedException: This request is not authorized to perform this operation using this permission.
RequestId:ffe36240-d01e-000a-41c5-aa22b5000000
Time:2023-06-29T20:06:46.5394329Z
Status: 403 (This request is not authorized to perform this operation using this permission.)
ErrorCode: AuthorizationPermissionMismatch

Content:
<?xml version="1.0" encoding="utf-8"?><Error><Code>AuthorizationPermissionMismatch</Code><Message>This request is not authorized to perform this operation using this permission.
RequestId:ffe36240-d01e-000a-41c5-aa22b5000000
Time:2023-06-29T20:06:46.5394329Z</Message></Error>

Headers:
Vary: Origin
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
x-ms-request-id: ffe36240-d01e-000a-41c5-aa22b5000000
x-ms-client-request-id: fe512045-1973-4eef-9dc6-c546015aea94
x-ms-version: 2022-11-02
x-ms-error-code: AuthorizationPermissionMismatch
Date: Thu, 29 Jun 2023 20:06:46 GMT
Content-Length: 279
Content-Type: application/xml

   at Azure.Storage.Blobs.<ListBlobHierarchySegmentAsync>d__59.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaiter.GetResult()
   +29 more...

Analyzing Assets ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/2159 Assets   0% 00:00:06 ⣻
                                                                                       Unhandled exception: System.AggregateException: An error occurred while writing to logger(s). (An item with the same key has already been added. Key: Vertical.SpectreLogger.Rendering.ExceptionRenderer+Options)
---> System.ArgumentException: An item with the same key has already been added. Key: Vertical.SpectreLogger.Rendering.ExceptionRenderer+Options
   at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
   at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value)
   at Vertical.SpectreLogger.Options.OptionsCollection.GetOptions[TOptions]()
   at Vertical.SpectreLogger.Rendering.ExceptionRenderer.Render(IWriteBuffer buffer, LogEventContext& context)
   at Vertical.SpectreLogger.Rendering.ExceptionRenderer.Vertical.SpectreLogger.Core.ITemplateRenderer.Render(IWriteBuffer buffer, LogEventContext& context)
   at Vertical.SpectreLogger.Rendering.RendererPipeline.Render(LogEventContext& logEventContext)
   at Vertical.SpectreLogger.Rendering.RendererPipeline.Vertical.SpectreLogger.Core.IRendererPipeline.Render(LogEventContext& logEventContext)
   at Vertical.SpectreLogger.SpectreLogger.Log[TState](LogLevel logLevel, EventId eventId, TState state, Exception exception, Func`3 formatter)
   at Microsoft.Extensions.Logging.Logger.<Log>g__LoggerLog|13_0[TState](LogLevel logLevel, EventId eventId, ILogger logger, Exception exception, Func`3 formatter, List`1& exceptions, TState& state)
   --- End of inner exception stack trace ---
   at Microsoft.Extensions.Logging.Logger.ThrowLoggingError(List`1 exceptions)
   at Microsoft.Extensions.Logging.Logger.Log[TState](LogLevel logLevel, EventId eventId, TState state, Exception exception, Func`3 formatter)
   at Microsoft.Extensions.Logging.Logger`1.Microsoft.Extensions.Logging.ILogger.Log[TState](LogLevel logLevel, EventId eventId, TState state, Exception exception, Func`3 formatter)
   at Microsoft.Extensions.Logging.LoggerExtensions.Log(ILogger logger, LogLevel logLevel, EventId eventId, Exception exception, String message, Object[] args)
   at Microsoft.Extensions.Logging.LoggerExtensions.Log(ILogger logger, LogLevel logLevel, Exception exception, String message, Object[] args)
   at Microsoft.Extensions.Logging.LoggerExtensions.LogError(ILogger logger, Exception exception, String message, Object[] args)
   at AMSMigrate.Ams.AssetAnalyzer.AnalyzeAsync(MediaAssetResource asset, BlobServiceClient storage, CancellationToken cancellationToken) in D:\z\azure-media-migration\ams\AssetAnalyzer.cs:line 98
   at AMSMigrate.Ams.AssetAnalyzer.<>c__DisplayClass5_0.<<MigrateAsync>b__1>d.MoveNext() in D:\z\azure-media-migration\ams\AssetAnalyzer.cs:line 146
--- End of stack trace from previous location ---
   at AMSMigrate.Ams.AssetAnalyzer.<>c__DisplayClass5_0.<<MigrateAsync>b__0>d.MoveNext() in D:\z\azure-media-migration\ams\AssetAnalyzer.cs:line 148
--- End of stack trace from previous location ---
   at AMSMigrate.Ams.BaseMigrator.MigrateInBatches[T](AsyncPageable`1 pageable, List`1 filteredList, Func`2 processBatch, Int32 batchSize, CancellationToken cancellationToken) in D:\z\azure-media-migration\ams\BaseMigrator.cs:line 61
   at AMSMigrate.Ams.BaseMigrator.MigrateInBatches[T](AsyncPageable`1 pageable, List`1 filteredList, Func`2 processBatch, Int32 batchSize, CancellationToken cancellationToken) in D:\z\azure-media-migration\ams\BaseMigrator.cs:line 57
   at AMSMigrate.Ams.AssetAnalyzer.MigrateAsync(CancellationToken cancellationToken) in D:\z\azure-media-migration\ams\AssetAnalyzer.cs:line 142
   at AMSMigrate.Program.AnalyzeAssetsAsync(GlobalOptions globalOptions, AnalysisOptions analysisOptions, CancellationToken cancellationToken) in D:\z\azure-media-migration\Program.cs:line 149
   at AMSMigrate.Program.<>c__DisplayClass0_0.<<Main>b__0>d.MoveNext() in D:\z\azure-media-migration\Program.cs:line 40
--- End of stack trace from previous location ---
   at System.CommandLine.Invocation.AnonymousCommandHandler.InvokeAsync(InvocationContext context)
   at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_0.<<BuildInvocationChain>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass17_0.<<UseParseErrorReporting>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass12_0.<<UseHelp>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass22_0.<<UseVersionOption>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass19_0.<<UseTypoCorrections>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseSuggestDirective>b__18_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass16_0.<<UseParseDirective>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<RegisterWithDotnetSuggest>b__5_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass8_0.<<UseExceptionHandler>b__0>d.MoveNext()

this is the cmaf muxer. depends on #107

Make sure the AmsMigrate.exe tool can run in multiple places and the data in the output folder is not messed up even if multiple instance of the tool run on the same input asset.

tracking https://msazure.visualstudio.com/One/_workitems/edit/24354392

Set the correct content type of the file being uploaded.

Current we log TRACE level to log, the log can grow to quite large. Open tracking issue to consider whether we need to address this issue or not...

@Prakash,

When running this command,

AMSMigrate.exe assets -s b2c12710-ecdf-4532-9ba6-7e74a219b5f2 -g pohhsuTest -n pohhsumediaservice -o https://pohhsustoragetest.blob.core.windows.net/ -t test-output/${AssetName}-stream-dash --skip-migrated false

I noticed that the progress report is reporting 40 Assets

Which is not correct…

If I do analyze, I see 24 assets

D:\z\azure-media-migration\bin\Debug\net6.0>AMSMigrate.exe analyze -s b2c12710-ecdf-4532-9ba6-7e74a219b5f2 -g pohhsuTest -n pohhsumediaservice
[1:26:02 PM Info] Begin analysis of assets for account: pohhsumediaservice
Asset Summary
┌─────────────┬────┐
│ Total │ 24 │
│ Streamable │ 0 │
│ Migrated │ 5 │
│ Failed │ 2 │
│ Skipped │ 0 │
│ No locators │ 24 │
└─────────────┴────┘

Which matches what I see in AMSE as well…

I see the value for Total Asset is from the ‘AssetCount’ metrics of AMS, and it is indeed returning 40, so no idea how/why this number was returned by AMS…

So the question is, what should we do here? Leave it alone? Fix by doing what analyze does? Or remove Total?

Thanks,
John

allow for

1. ismv -> cmaf allows preservation of timestamps, could be useful for fixing gaps
2. ismv parser is needed for ismv -> vtt later on.

this work will enable #108 , #109 (cmaf output, vtt output)

Expression encoder incorrectly sets the version of the 'trun' box to be 0 (instead of 1). So shaka packager treats the composition time as large positive number instead of negative.

Possible fixes.1

1. Modify the fragmented MP4 and update the trun version to 1.
2. Check if ffmpeg can handle it and use ffmpeg to transmux the content.

Some assets (VOD or Live) that were created with V2 APIs might have been encrypted with a specific content key. The migration tool would also support such asset as input and make the asset streaming-able through DASH/HLS.

Issue:

Running this command (with the highlighted pattern)

AMSMigrate.exe assets -s b2c12710-ecdf-4532-9ba6-7e74a219b5f2 -g pohhsuTest -n pohhsumediaservice -o pohhsumediaservice -t test-output-1/${AssetName}-stream-dash -f "name eq '0365video-03f3f58678-StandardEncoder-H265ContentAwareEncoding-042d34d192'"

Fails with this:

[11:05:23 AM Fail] Failed to migrate asset 0365video-03f3f58678-StandardEncoder-H265ContentAwareEncoding-042d34d192 Error:Azure.RequestFailedException: The specified container does not exist.
RequestId:038ac1db-401e-000d-4434-a564e9000000━━━━━━━━━━ 0/40 Assets 0% 00:00:55 ⣟
Time:2023-06-22T18:04:35.7213969Z
Status: 404 (The specified container does not exist.)
ErrorCode: ContainerNotFound

Content:

RequestId:038ac1db-401e-000d-4434-a564e9000000
Time:2023-06-22T18:04:35.7213969Z

Headers:
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
x-ms-request-id: 038ac1db-401e-000d-4434-a564e9000000
x-ms-client-request-id: b9006f32-e432-494b-9ada-14267d810579
x-ms-version: 2022-11-02
x-ms-error-code: ContainerNotFound
Date: Thu, 22 Jun 2023 18:04:34 GMT
Content-Length: 225
Content-Type: application/xml

at Azure.Storage.Blobs.BlobRestClient.StartCopyFromURLAsync(String copySource, Nullable1 timeout, IDictionary2 metadata, Nullable1 tier, Nullable1 rehydratePriority, Nullable1 sourceIfModifiedSince, Nullable1
sourceIfUnmodifiedSince, String sourceIfMatch, String sourceIfNoneMatch, String sourceIfTags, Nullable1 ifModifiedSince, Nullable1 ifUnmodifiedSince, String ifMatch, String ifNoneMatch, String ifTags, String leaseId, String
blobTagsString, Nullable1 sealBlob, Nullable1 immutabilityPolicyExpiry, Nullable1 immutabilityPolicyMode, Nullable1 legalHold, CancellationToken cancellationToken)
at Azure.Storage.Blobs.Specialized.BlobBaseClient.StartCopyFromUriInternal(Uri source, IDictionary2 metadata, IDictionary2 tags, Nullable1 accessTier, BlobRequestConditions sourceConditions, BlobRequestConditions destinationConditions, Nullable1 rehydratePriority, Nullable1 sealBlob, BlobImmutabilityPolicy destinationImmutabilityPolicy, Nullable1 legalHold, Boolean async, CancellationToken cancellationToken)
at Azure.Storage.Blobs.Specialized.BlobBaseClient.StartCopyFromUriAsync(Uri source, IDictionary2 metadata, Nullable1 accessTier, BlobRequestConditions sourceConditions, BlobRequestConditions destinationConditions, Nullable1 rehydratePriority, CancellationToken cancellationToken) at AMSMigrate.Azure.AzureStorageUploader.UploadBlobAsync(String containerName, String fileName, BlockBlobClient blob, CancellationToken cancellationToken) in D:\z\azure-media-migration\azure\AzureStorageUploader.cs:line 67 at AMSMigrate.Transform.StorageTransform.UploadBlobAsync(BlockBlobClient blob, ValueTuple2 outputPath, CancellationToken cancellationToken) in D:\z\azure-media-migration\transform\StorageTransform.cs:line 85
at AMSMigrate.Transform.PackageTransform.<>c__DisplayClass3_0.<b__1>d.MoveNext() in D:\z\azure-media-migration\transform\PackageTransform.cs:line 60
--- End of stack trace from previous location ---
at AMSMigrate.Transform.PackageTransform.TransformAsync(AssetDetails details, ValueTuple`2 outputPath, CancellationToken cancellationToken) in D:\z\azure-media-migration\transform\PackageTransform.cs:line 128

Reason:

_Right now it is missing one line in code.
Adding await container.CreateIfNotExistsAsync() to AzureStorageUplaoder.cs would fix this.
But I wasn’t sure if we want to do this for every single file or if we want to optimize it by remembering the container created so if uploading to same container, we only need to call this once.

-Prakash._

ID3 or other metadata that the AMS origin adds to the video track in the emsg boxes of a live stream will not be preserved by the conversion process. Similarly any other out of band data that's only present in the output manifest such as SCTE signals will not be preserved.

From assets

• remove --delete-migrated (to be replaced by separate cleanup command)
• remove {LocatorId} from path-template.
• remove --mark-complete

D:\z\azure-media-migration\bin\Debug\net6.0>AMSMigrate.exe assets -s b2c12710-ecdf-4532-9ba6-7e74a219b5f2 -g pohhsuTest -n pohhsumediaservice -o pohhsumediaservice -t test-output/${AssetName}-stream-dash -f "name eq 'onznet-a6634ccfb6'" --skip-migrated False --copy-nonstreamable False

[7:22:57 PM Info] Begin migration of assets for account: pohhsumediaservice
[7:23:01 PM Info] Migrating asset: onznet-a6634ccfb6 ...
[7:23:05 PM Warn] No manifest (.ism file) found in container onznet-a6634ccfb6
[7:23:05 PM Info] Finished migration of assets for account: pohhsumediaservice. Time taken: 00:00:17.4252045
┌──────────────────┬───────┐
│ Asset Type       │ Count │
├──────────────────┼───────┤
│ Total            │ 1     │
│ Already Migrated │ 0     │
│ Skipped          │ 0     │
│ Successful       │ 0     │
│ Failed           │ 1     │     <----------------------------------- shouldn't be failed.
│ Deleted          │ 0     │
└──────────────────┴───────┘
[7:23:05 PM Info] See file D:\z\azure-media-migration\bin\Debug\net6.0\MigrationLog_19_22_47.txt for detailed logs.

We can enable it in the future if customers ask for it.

It is a regression of previous code refactoring to make "storage" and "assets" command hold their own command line optins.

When AMSMigrate.exe analyze command runs, it consumes the metadata settings of each asset container.

When AMSMigrate.exe assets or storage command runs, it updates the metadata settings for the related assest container.

tracking: https://msazure.visualstudio.com/One/_workitems/edit/24354291

The leasing process creates a __migrate blob and that is not delete after migration is complete

Prakash, I'm not sure if this is a known issue (related to issue #24).

But I'm hitting hang for this command line:

./AMSMigrate assets -s b2c12710-ecdf-4532-9ba6-7e74a219b5f2 --skip-migrated False -g pohhsuTest -n pohhsumediaservice -o pohhsumediaservice -t test-output/${AssetName}-stream-dash -f "name eq '0365video-03f3f58678-StandardEncoder-H265ContentAwareEncoding-042d34d192'"

[19:56:57 Dbug] Writing logs to /home/pohhsu/z/azure-media-migration/bin/Debug/net6.0/MigrationLog_19_56_57.txt
[19:57:09 Info] Begin migration of assets for account: pohhsumediaservice
[19:57:10 Info] The total asset count of the media account is 24.
[19:57:10 Info] The total assets to handle in this run is 1.
[19:57:10 Info] Migrating asset: 0365video-03f3f58678-StandardEncoder-H265ContentAwareEncoding-042d34d192 (container asset-751c816a-0b82-4914-ac1d-832d28e759b4) ...
[19:57:13 Dbug] Starting upload of /tmp/AMSMigrate/0365video-03f3f58678-StandardEncoder-H265ContentAwareEncoding-042d34d192/output/O365Video_Azure1080p_0.mp4 to storage O365Video_Azure1080p_0.mp4
[19:57:13 Dbug] Starting upload of /tmp/AMSMigrate/0365video-03f3f58678-StandardEncoder-H265ContentAwareEncoding-042d34d192/output/O365Video_Azure1080p_1.mp4 to storage O365Video_Azure1080p_1.mp4

Asset Migration ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/1 Assets   0% 00:05:07 ⣾

Note that the process can't be killed once hung with ctrl+c.
On a side note, Doing kill -9 works, but will leave the pipe file opened, so subsequent run will fail in createpipe with this error:

Error:System.InvalidOperationException: failed to create pipe: wareEncoding-042d34d192/O365Video
/tmp/AMSMigrate/0365video-03f3f58678-StandardEncoder-H265ContentAwareEncoding-042d34d192/O365Video_Azure1080p_1280x720_AACAudio_246.mp4

which can be fixed manually by delete this file / folder...

Below options --delete-migrated, --mark-complete will be removed from "assets" and "storage" command

Execute the following command and return 3 results:
AMSMigrate.exe analyze -s 2b461b25-f7b4-4a22-90cc-d640a14b5471 -g johndeu_DO_NOT_DELETE -n nimbuspm -cs 2022-05-18T23:00:00Z -ce 2022-05-19T23:00:00Z

Asset Summary

┌─────────────┬───┐

│ Total │ 3 │

│ Streamable │ 3 │

│ Migrated │ 3 │

│ Failed │ 0 │

│ Skipped │ 0 │

│ No locators │ 3 │

└─────────────┴───┘
Breaking the query into three parts yields zero results:
AMSMigrate.exe analyze -s 2b461b25-f7b4-4a22-90cc-d640a14b5471 -g johndeu_DO_NOT_DELETE -n nimbuspm -cs 2022-05-18T23:00:00Z -ce 2022-05-18T23:59:59Z
AMSMigrate.exe analyze -s 2b461b25-f7b4-4a22-90cc-d640a14b5471 -g johndeu_DO_NOT_DELETE -n nimbuspm -cs 2022-05-19T00:00:00Z -ce 2022-05-19T12:00:00Z
AMSMigrate.exe analyze -s 2b461b25-f7b4-4a22-90cc-d640a14b5471 -g johndeu_DO_NOT_DELETE -n nimbuspm -cs 2022-05-19T12:00:00Z -ce 2022-05-19T23:00:00Z

Asset Summary

┌─────────────┬───┐

│ Total │ 0 │

│ Streamable │ 0 │

│ Migrated │ 0 │

│ Failed │ 0 │

│ Skipped │ 0 │

│ No locators │ 0 │

└─────────────┴───┘

Upon initial investigation, it is evident that the three assets have been created on:
{5/18/2022 11:19:46 PM +00:00}
{5/18/2022 5:47:39 PM +00:00}
{5/18/2022 6:03:57 PM +00:00}

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request

Handle the situation when ams account has multiple storage accounts, and the asset is not in the primary storage account.
update the following code looping through all containers
if (!await container.ExistsAsync(cancellationToken))
{
...
return false;
}

The expected behavior is to copy over files that are not described in the .ism file to the destination container.

For those files that are described in the .ism file, the Shaka packager is responsible for generating cmaf files plus .mpd/.m3u8 and uploading to the destination container, the original media files are not supposed to be put in the destination folder.

./AMSMigrate assets -s 2b461b25-f7b4-4a22-90cc-d640a14b5471 -g johndeu_DO_NOT_DELETE -n nimbuspm -o amsencodermsitest -t ${AssetName} -f "name eq 'dd567de1-dbea-4e8a-9357-95e35bc6a33c'"
[11:06:23 AM Dbug] Writing logs to /mnt/d/GitHub/azure-media-migration/publish/MigrationLog_11_06_22.txt
[11:06:26 AM Info] Begin migration of assets for account: nimbuspm
[11:06:26 AM Info] The total asset count of the media account is 2159.
[11:06:26 AM Info] The total assets to handle in this run is 1.
[11:06:27 AM Info] Migrating asset: dd567de1-dbea-4e8a-9357-95e35bc6a33c (container
asset-dd567de1-dbea-4e8a-9357-95e35bc6a33c) ...━━━━━━━━━ 0/1 Assets 0% 00:00:00 ⣟
[11:06:28 AM Fail] Failed to migrate asset dd567de1-dbea-4e8a-9357-95e35bc6a33c.
Azure.RequestFailedException: Value for one of the query parameters specified in the request URI is invalid.
RequestId:8c0c1049-001e-003b-0622-b4c469000000
Time:2023-07-11T18:06:28.3208087Z
Status: 400 (Value for one of the query parameters specified in the request URI is invalid.)
ErrorCode: InvalidQueryParameterValue

Additional Information:
QueryParameterName: comp
QueryParameterValue:
Reason:

Content:
InvalidQueryParameterValueValue for one of the
query parameters specified in the request URI is invalid.
RequestId:8c0c1049-001e-003b-0622-b4c469000000
Time:2023-07-11T18:06:28.3208087Zcomp

Headers:
Server: Microsoft-HTTPAPI/2.0
x-ms-request-id: 8c0c1049-001e-003b-0622-b4c469000000
x-ms-client-request-id: 271646c7-00a0-4e36-88fb-0f95d2129710
x-ms-error-code: InvalidQueryParameterValue
Date: Tue, 11 Jul 2023 18:06:27 GMT
Content-Length: 351
Content-Type: application/xml

at Azure.Storage.Blobs.d__8.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(
System.Threading.Tasks.Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaiter.GetResult()
+26 more...
[11:06:28 AM Dbug] Migrated asset: dd567de1-dbea-4e8a-9357-95e35bc6a33c, container:
asset-dd567de1-dbea-4e8a-9357-95e35bc6a33c, type: mp4-v2, status: Failed 00:00:01 ⣾
[11:06:28 AM Info] Finished migration of assets for account: nimbuspm. Time taken: 00:00:03.9083111
┌──────────────────┬───────┐
│ Asset Type │ Count │
├──────────────────┼───────┤
│ Total │ 1 │
│ Already Migrated │ 0 │
│ Skipped │ 0 │
│ Successful │ 0 │
│ Failed │ 1 │
└──────────────────┴───────┘

A commd to qucikly reset all the assets migration status.

Issue: @BruceLin-MS hit exception when running AMSMigrate. The exception says "Please run 'az login' to set up account". After doing az login, then everything is fine. From discussion with Prakash, originally the tool enabled browser prompt but it wasn't working for WSL2 and a bit difficult for running on the cloud. It might be easiest to just require and document that the user must do az login. Also, a consideration is that adding 'Storage Blob Delegator' role to storage might be easier to document if we give the az command. So we need to make a decision on this.

@ningl2, what do you think we should do here?

The code always uses the total account of assets or container under storage account to report the progress.

When a filter is added to the command line, the actual assets for handling would be much smaller.

Right now the default log is a very verbose log. Useful for troubleshooting but hard for summarizing. Logging just the failures to a separate log file would make it easy to quickly summarize failures.

AMSMigrate.exe analyze -n works with ams-account
D:\Packager\azure-media-migration\bin\Debug\net6.0>AMSMigrate.exe analyze -s 2b461b25-f7b4-4a22-90cc-d640a14b5471 -g johndeu_DO_NOT_DELETE -n nimbuspm

but doesn't work for storage account
D:\Packager\azure-media-migration\bin\Debug\net6.0>AMSMigrate.exe analyze -s 2b461b25-f7b4-4a22-90cc-d640a14b5471 -g johndeu_DO_NOT_DELETE -n nimbuspmteam

Design and implement clean up. (account?, migrated assets, etc)

When running analyze the count of streamable assets is incorrect.

Add AssetType, Manifest in HTML report and make 'Report' as default analysis type

Take step to open source azure-media-migration