julianhayward / azapicall Goto Github PK

Hey,
some of our users have access to the Azure Portal but don't have RBAC permissions on a subscription/resourcegroup/resource.
Regarding to this, they can authenticate with Connect-AzAccount without having a context.
Nevertheless, they also want to be able to use the AzAPICall.

Error message:
AzAPICall.ps1: You cannot call a method on a null-valued expression.

Debugger show that this is related to the following rows (218-243):

#API Call Tracking
$tstmp = (Get-Date -format "yyyyMMddHHmmssms")
$null = $script:arrayAPICallTracking.Add([PSCustomObject]@{ 
        CurrentTask                          = $currentTask
        TargetEndpoint                       = $targetEndpoint
        Uri                                  = $uri
        Method                               = $method
        TryCounter                           = $tryCounter
        TryCounterUnexpectedError            = $tryCounterUnexpectedError
        RetryAuthorizationFailedCounter      = $retryAuthorizationFailedCounter
        RestartDueToDuplicateNextlinkCounter = $restartDueToDuplicateNextlinkCounter
        TimeStamp                            = $tstmp
    })

if ($caller -eq "CustomDataCollection") {
    $null = $script:arrayAPICallTrackingCustomDataCollection.Add([PSCustomObject]@{ 
            CurrentTask                          = $currentTask
            TargetEndpoint                       = $targetEndpoint
            Uri                                  = $uri
            Method                               = $method
            TryCounter                           = $tryCounter
            TryCounterUnexpectedError            = $tryCounterUnexpectedError
            RetryAuthorizationFailedCounter      = $retryAuthorizationFailedCounter
            RestartDueToDuplicateNextlinkCounter = $restartDueToDuplicateNextlinkCounter
            TimeStamp                            = $tstmp
        })
}

In my branch I tested and added the following lines which will init the mentioned variabels:

$arrayAPICallTracking = [System.Collections.ArrayList]@()
$arrayAPICallTrackingCustomDataCollection = [System.Collections.ArrayList]@()

or if you would like to run the powershell function in parallel:

$arrayAPICallTracking = [System.Collections.ArrayList]::Synchronized((New-Object System.Collections.ArrayList))
$arrayAPICallTrackingCustomDataCollection = [System.Collections.ArrayList]::Synchronized((New-Object System.Collections.ArrayList))

The ConsistencyLevel header required for advanced queries against directory objects is not included by default in subsequent page requests. It must be set explicitly in subsequent pages.

Regarding to this, an additional parameter for the "AzAPICall"-function call would be needed.

Within the new product "AzAdServicePrincipalInsights" the usage of "consistencyLevel" is already given:
https://github.com/JulianHayward/AzADServicePrincipalInsights/blob/main/pwsh/AzADServicePrincipalInsights.ps1

Example of listing more than 100 private DNS zones will stuck in an endless loop:
https://docs.microsoft.com/en-us/rest/api/dns/privatedns/privatezones/list

Initial requests:

Next link request:

Unfortunately, a ":443" is within the NextLink response and then won't match the "$arrayAzureManagementEndPointUrls":

if ($arrayAzureManagementEndPointUrls | Where-Object { $uri -match $_ }) {
    $targetEndpoint = "ManagementAPI"
    $bearerToUse = $htBearerAccessToken.AccessTokenManagement
}
else {
    $targetEndpoint = "MSGraphAPI"
    $bearerToUse = $htBearerAccessToken.AccessTokenMSGraph
}

Regarding to this, no bearer token will be used and the variable "$bearerToUse" is empty.
The request will got a authorization error regarding the missing bearer token.

Quick solution is to check and replace the ":443":

if ($azAPIRequestConvertedFromJson.nextLink) {
  $isMore = $true
  if ($uri -eq $azAPIRequestConvertedFromJson.nextLink) {
      if ($restartDueToDuplicateNextlinkCounter -gt 3) {
          Write-Host " $currentTask restartDueToDuplicateNextlinkCounter: #$($restartDueToDuplicateNextlinkCounter) - Please report this error/exit"
          Throw "Error - check the last console output for details"
      }
      else {
          $restartDueToDuplicateNextlinkCounter++
          Write-Host "nextLinkLog: uri is equal to nextLinkUri"
          Write-Host "nextLinkLog: uri: $uri"
          Write-Host "nextLinkLog: nextLinkUri: $($azAPIRequestConvertedFromJson.nextLink)"
          Write-Host "nextLinkLog: re-starting (#$($restartDueToDuplicateNextlinkCounter)) '$currentTask'"
          $apiCallResultsCollection = [System.Collections.ArrayList]@()
          $uri = $initialUri
          Start-Sleep -Seconds 1
          createBearerToken -targetEndPoint $targetEndpoint
          Start-Sleep -Seconds 1
      }
  }
  else {
      $uri = $azAPIRequestConvertedFromJson.nextLink
      if($uri -match ":443"){
          $uri = $uri -replace ":443"
      }
  }
  if ($htParameters.DebugAzAPICall -eq $true) { Write-Host "   DEBUG: nextLink: $Uri" -ForegroundColor $debugForeGroundColor }
  }

While the Check AAD UserType an unhandled error happens:
StatusCode: 401 (Unauthorized)

If the user just call the API with the parameters of the Check AAD UserType-function it is working as expected:

The user has access to the AzureAD but not to resources.

Now, we would like to only use the Azure APIs instead of AZ cmdlets.
Regarding to this, several APIs will be tested.

One, which isn't working in the main branch, is checking the existence of an resource group:
https://docs.microsoft.com/en-us/rest/api/resources/resourcegroups/checkexistence

As explained in the article the statuscode (204) of the API call or the statuscode (404 => NotFound) of the error in the catch is needed.
Regarding to this, an add $listenOn-Parameter like "StatusCode" is needed.
Additionally, not only the status code 200 is accepted and within the catch-part of the API call a status code check is needed as well.

Within the branch "user/kaiaschulz/mgmtapi-rg_check" a first test was successful.

Hey,
by default, the Microsoft Graph API respond with just 100 entries.
To increase the result entries we can use $top.
$top value must be between 1 and 999 inclusive.
The query parameter "$top" can use paging: https://docs.microsoft.com/en-us/graph/paging

If I use the query parameter "$top=50" should be used for testing purposes the respond will include a NextLink for paging.
Regarding to this, the "AzAPICall"-function will iterate further.

To prevent this behavior I would suggest to add another "Paging"-parameter to the "AzAPICall"-function.
If "Paging"-parameter is $true, then it should proceed further with the received "@oData.nextLink":

old:

elseIf ($azAPIRequestConvertedFromJson."@oData.nextLink") {

new:

elseIf ($azAPIRequestConvertedFromJson."@oData.nextLink" -and $Paging) {

$script:azAPICallConf = initAzAPICall

$uri = "$($azAPICallConf['azAPIEndpointUrls'].MicrosoftGraph)/beta/groups/<GroupId>/members/`$ref"
$uri

$Body = "{ ""@odata.id"": ""$($azAPICallConf['azAPIEndpointUrls'].MicrosoftGraph)/beta/directoryObjects/<EnterpriseApplicationObjectId>" }"
$Body

$AddAzureADGroupMember = AzAPICall -uri $uri -AzAPICallConfiguration $azAPICallConf -method 'POST' -body $Body
$AddAzureADGroupMember

POST https://microsoftgraph.chinacloudapi.cn/beta/groups/_<GroupId>_/members/$ref - try #1; returned: (StatusCode: '204') <.code: ''> <.error.code: ''> | <.message: ''> <.error.message: ''> - (plain : ) try again in 7 second(s)

https://docs.microsoft.com/en-us/graph/api/group-post-members?view=graph-rest-1.0&tabs=http
The status code of the request is 204. Regarding to this, the following code need to be replaced:

if ($azAPIRequest.StatusCode -ne 200) {

if ($azAPIRequest.StatusCode -notin 200..204) {

Add parameter "UnhandledErrorAction" to AzApiCall and AzAPICallErrorHandler functions and to update Documentation as such.

UnhandledErrorAction Parameter will determine how the AzAPICallErrorHandler function is exited when an Unhandled exception occurs

Parameter UnhandledErrorAction will accept values "Stop" (default) & "Continue" and aligns with PowerShell's ErrorAction values "Stop" and "Continue"

The UnhandledErrorAction Parameter Values are:
"Stop" will end processing with Throw 'Error - check the last console output for details'
"Continue" will break to return and continue processing

The Following to be added to the Param() section in both functions:

[ValidateSet('Stop','Continue')]
[string] $UnhandledErrorAction = 'Stop'

AzAPICall is causing an endless loop with an internal server error:

AzAPICall -uri $Uri -AzAPICallConfiguration $azAPICallConf -Method $method -currentTask $currentTask -body $body -listenOn 'Content'

[AzAPICall] DEBUGTASK: exportTemplate -> unexpectedError: false
[AzAPICall] DEBUGTASK: exportTemplate -> apiStatusCode: '500' (InternalServerError)
[AzAPICallErrorHandler] exportTemplate try #345; return: (StatusCode: '500' (InternalServerError)) <.code: ''> <.error.code: 'InternalServerError'> | <.message: ''> <.error.message: 'Encountered internal server error. Diagnostic information: timestamp '20221216T074356Z', subscription id 'xxxxxxxx, tracking id 'xxxxxxxx, request correlation id 'xxxxxxxx'.'> - AzAPICall: try again in 345 second(s)
[AzAPICall] DEBUGTASK: exportTemplate -> [AzAPICall] exportTemplate - retry
[AzAPICall] DEBUGTASK: exportTemplate -> attempt#346 processing: exportTemplate uri: 'https://management.azure.com/subscriptions/xxxxxxxx/resourceGroups/xxxxxxxx/exportTemplate?api-version=2019-05-01' body: '{
"resources": [
    "*"
],
"options": "IncludeParameterDefaultValue"
 }
'

implement telemetry functionality to get information about usage

When trying to use the property -listenOn 'Headers' the default will response with 'value' instead:

 [AzAPICall] DEBUGTASK: exportTemplate - resourceGroupId: /subscriptions/xxx/resourceGroups/yyy, resourceId: * -> attempt#1 processing: exportTemplate - resourceGroupId: /subscriptions/xxx/resourceGroups/yyy, resourceId: * uri: 'https://management.azure.com/subscriptions/xxx/resourceGroups/yyy/exportTemplate?api-version=2019-10-01' body: '{
    "resources": [
        "*"
    ],
    "options": "IncludeParameterDefaultValue"
}
'
  [AzAPICall] DEBUGTASK: exportTemplate - resourceGroupId: /subscriptions/xxx/resourceGroups/yyy, resourceId: * -> unexpectedError: false
  [AzAPICall] DEBUGTASK: exportTemplate - resourceGroupId: /subscriptions/xxx/resourceGroups/yyy, resourceId: * -> apiStatusCode: '202' (OK)
  [AzAPICall] DEBUGTASK: exportTemplate - resourceGroupId: /subscriptions/xxx/resourceGroups/yyy, resourceId: * -> listenOn=Headers (13)
  [AzAPICall] DEBUGTASK: exportTemplate - resourceGroupId: /subscriptions/xxx/resourceGroups/yyy, resourceId: * -> listenOn=Default(Value) value not exists; return empty array
  [AzAPICall] DEBUGTASK: exportTemplate - resourceGroupId: /subscriptions/xxx/resourceGroups/yyy, resourceId: * -> NextLink/skipToken/NextMarker: none

In the specific case of exporting templates (ARM) we have several statusCode which are responding:
https://learn.microsoft.com/en-us/rest/api/resources/resource-groups/export-template?tabs=HTTP#response

In this case, if the status code is 200 the content will be filled.
If the status code is 202 instead, within the header.location a new uri will be available as feedback which need to be used to send the request again.

Actually, the if-clause for the listenOn isn't set correctly.

In some combinations of the writeMethod and debugWriteMethod the initialization of AzAPICall is failing and running into an error:

Line |
1788 |  … rs'] += setHtParameters -AzAccountsVersion $AzAccountsVersion -gitHub …
     |                                               ~~~~~~~~~~~~~~~~~~
     | Cannot process argument transformation on parameter 'AzAccountsVersion'. Cannot convert value to type System.String.
  AzAPICall htParameters:

Name             Value
----             -----
writeMethod      Output
debugWriteMethod Output


  Create htParameters succeeded
 Get Az context
  Get Az context succeeded
WARNING:  Check Az context
WARNING:   Az context AccountId: ''
WARNING:   Az context AccountType: ''
InvalidOperation: C:\<Path>\Modules\AzAPICall\1.1.68\functions\AzAPICallFunctions.ps1:1817:61
Line |
1817 |  … accountType = $($AzAPICallConfiguration['checkContext'].Account.Type)
     |                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | The property 'accountType' cannot be found on this object. Verify that the property exists and can be set.

WARNING:   Check Az context failed: Az context is not set to any Subscription
WARNING:   Set Az context to a subscription by running: Set-AzContext -subscription <subscriptionId> (run Get-AzSubscription to get the list of available Subscriptions). When done re-run the script
WARNING:   OR
WARNING:   Use parameter -SubscriptionId4AzContext - e.g. .\AzGovVizParallel.ps1 -SubscriptionId4AzContext <subscriptionId>
Exception: C:\<Path>\Modules\AzAPICall\1.1.68\functions\AzAPICallFunctions.ps1:1859:9
Line |
1859 |          Throw 'Error - check the last console output for details'
     |          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | Error - check the last console output for details

Regarding to this, I wrote a small script to analyze which of the possible 64 combinations are affected to identify them and improving the code:

The result of the tests shows that 21 combinations of the 64 test cases are affected:

1. Write-Error: writeMethod: Error; debugWriteMethod: Debug
2. Write-Error: writeMethod: Error; debugWriteMethod: Information
3. Write-Error: writeMethod: Error; debugWriteMethod: Error
4. Write-Error: writeMethod: Error; debugWriteMethod: Output
5. Write-Error: writeMethod: Error; debugWriteMethod: Verbose
6. Write-Error: writeMethod: Error; debugWriteMethod: Warning
7. Write-Error: writeMethod: Error; debugWriteMethod: Host
8. Write-Error: writeMethod: Error; debugWriteMethod: Progress
9. Write-Error: writeMethod: Output; debugWriteMethod: Output
10. Write-Error: writeMethod: Output; debugWriteMethod: Debug
11. Write-Error: writeMethod: Output; debugWriteMethod: Verbose
12. Write-Error: writeMethod: Output; debugWriteMethod: Information
13. Write-Error: writeMethod: Output; debugWriteMethod: Progress
14. Write-Error: writeMethod: Output; debugWriteMethod: Warning
15. Write-Error: writeMethod: Output; debugWriteMethod: Host
16. Write-Error: writeMethod: Debug; debugWriteMethod: Error
17. Write-Error: writeMethod: Progress; debugWriteMethod: Error
18. Write-Error: writeMethod: Verbose; debugWriteMethod: Error
19. Write-Error: writeMethod: Warning; debugWriteMethod: Error
20. Write-Error: writeMethod: Information; debugWriteMethod: Error
21. Write-Error: writeMethod: Host; debugWriteMethod: Error

The different methods of writing are needed for a customization of the output.
Some of the Azure services, e.g. Automation Account, won't display Write-Host.

I cloned the Repo using VS Code to C:\Git_Ext
In my script, I did a:
Import-Module -Scope Local -Name 'C:\Git_Ext\AzAPICall\AzApiCall\AzApiCall.psm1'
and got this error:

Get-PSFConfigValue : The term 'Get-PSFConfigValue' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct
and try again.

Do we have any instructions on how to build the Module so I can import locally vs from a gallery?

It seems that the re-try isn't working anymore:

Hi.

I use AzAPICall as part of AzGovViz.
Using AzGovViz, I export data from several tenants, however, one of the tenants suddenly required the SubscriptionId4AzContext, which I have then provided. However, the addition of SubscriptionId4AzContext makes a Set-AzContext fail with "Please provide a valid tenant or a valid subscription." as if it can't find the subscription.

AzAPICall tries to Set-AzContext at line 1834 and then throws. This could of course be because I give it a wrong subscription ID but just before the Set-AzContext it tests the subscription at line 1830, using testSubscription, which correctly finds the subscription. I also verified that the ID it checked is the correct one.

I have tried logging in in Azure using the app registration that I use for the pipeline and I can switch between all necessary subscriptions.
I can't find a reason why it will not set the context.

Hello,

While running the script I am getting this error.

PS C:\Users\Admin\Downloads> .\AzGovVizParallel.ps1 -ManagementGroupId bxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Security warning
Run only scripts that you trust. While scripts from the internet can be useful, this script can potentially harm your
computer. If you trust this script, use the Unblock-File cmdlet to allow the script to run without this warning
message. Do you want to run C:\Users\Admin\Downloads\AzGovVizParallel.ps1?
[D] Do not run [R] Run once [S] Suspend [?] Help (default is "D"): R
ParserError: C:\Users\Admin\Downloads\AzGovVizParallel.ps1:210
Line |
210 | … k Button--medium Button d-lg-none color-fg-inherit p-1"> <span cla …
| ~
| The '<' operator is reserved for future use.
PS C:\Users\Admin\Downloads>

Please refer to image too.

Regards,
WVDAdminTF

adding to createbearertoken the endpoint for keyvault

missing Cred in Github Secrets