ruiromano / pbimonitor Goto Github PK

View Code? Open in Web Editor NEW

215.0 215.0 78.0 13.06 MB

License: MIT License

PowerShell 100.00%

pbimonitor's Introduction

Hi 👋, I'm Rui

❤️ Sponsor

pbimonitor's People

Contributors

Stargazers

Watchers

Forkers

zachrenwick samaguire kavasimihaly timsnedden isantillan1 rajamaniv weiplanet ievsantillan sivamani-dev bennidejagere llenroc pierreddata wangrandk zloyshadow constancelee fpvmorais lennychao musicdendy sureshpaulraj wottabyte hrmartins kevarnold972 bernatagulloesbrina pauljenny mikkidos pauloricardoferreira mrtoyotatmh andreeadan27 joelles29 iourichadour tom-juntunen sanjufl mchebihi rajaprabus27 numanthugormen gorgoras nexus150 cproudfoot92 lm0007 renegorismsft dgosbell stuartdickenson skimiforow nsi-dwaine stbere cinshiqaq ndh150 bilupa l33smyth aegonsql lkraehen xiong416 sergio-devscope vbennettv alanflorance andrewcchoi rdurbano willcisler loahcodes brenoneto12 johndev13 j-csy henshinsightlab rkapp22 cdrguru jamesdbartlett3 annamallekbojketr igthub cuongthh pbiqueryous ruthraraj17 tutuldot flederflick webmasterdevlin verdane trymzet

pbimonitor's Issues

Problem with Fetch - Catalog.ps1

When i run the script i get error below:

Get-ArrayInBatches : The term 'Get-ArrayInBatches' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the
path is correct and try again.
At C:\Git\pbimonitor\Fetch - Catalog.ps1:199 char:9

    Get-ArrayInBatches -array $workspacesModified -label "GetInfo ...

```
    ~~~~~~~~~~~~~~~~~~
```
- CategoryInfo : ObjectNotFound: (Get-ArrayInBatches:String) [], CommandNotFoundException
- FullyQualifiedErrorId : CommandNotFoundException

What does this mean? It seems a function is not declared?

Where do I find the config file to modify?

Sorry for the noob question, but where do I find the config file that must be modified?
Have run the install module command, but don't find any new programs installed where I would usually search for this file.
Could it be that the install has failed without a notification?

ActivityEvents are returning blanks

When executing all of the steps provided everything appears to work except for the ActivityEvents portion. The set of json files for Activities are fairly large so I can tell that it's finding data, but the outputs appear to be empty when polling details with repeating rows like this over and over again:

[{"Id":[],"RecordType":[],"CreationTime":[],"Operation":[],"OrganizationId":[],"UserType":[],"UserKey":[],"Workload":[],"UserId":[],"ClientIP":[],"UserAgent":[],"Activity":[],"ItemName":[],"WorkSpaceName":[],"CapacityId":[],"CapacityName":[],"WorkspaceId":[],"ObjectId":[],"DataflowId":[],"DataflowName":[]

The document doesn't list any prerequesites for this portion of the process as far as permissions, but I know that the activity events api requires this:

Global admins and Power BI service admins have access.

Is there anything that could be missing possibly?

Deleted reports still showing

First of all, really appreciate this solution. Overall it works well, but as soon as I delete a report, it will still show in the Power BI monitor report 2 weeks after (the refresh was done and all other things seem to show well). Does it take more time to let these reports disappear or might there be another reason?

GraphExtractGroups config setting is ignored

The config contains a line to (de)activate the extraction of group members:
"GraphExtractGroups": "false"
The script "Fetch - Graph.ps1" is testing the value, eg on line 155:
if ($config.GraphExtractGroups)
Whatever value is used in the config file, it always triggers the above if statement. It is likely interpreted as a string value instead of Boolean.

Azure function issue

We are facing a duplicate key issue in requirements.psd1

Following is the error code:

Result: Failure Exception: Failed to install function app dependencies. Error: 'The script file 'requirements.psd1' has parsing errors: Duplicate keys 'MicrosoftPowerBIMgmt.Profile' are not allowed in hash literals. ' Stack: at Microsoft.Azure.Functions.PowerShellWorker.DependencyManagement.DependencyManager.Initialize(ILogger logger) in D:\a_work\1\s\src\DependencyManagement\DependencyManager.cs:line 122 at Microsoft.Azure.Functions.PowerShellWorker.DependencyManagement.DependencyManager.Initialize(StreamingMessage request, ILogger logger) in D:\a_work\1\s\src\DependencyManagement\DependencyManager.cs:line 94 at Microsoft.Azure.Functions.PowerShellWorker.RequestProcessor.ProcessFunctionLoadRequest(StreamingMessage request) in D:\a_work\1\s\src\RequestProcessor.cs:line 196

Bug in LastUsageDateTime calculated columns

The LastUsageDateTime calculated columns are supposed to retrieve the DateTime of the current item's Last Usage from the Activity table, but they only reference the Activity[Time] column, which only contains the time of day. When these calculated columns convert Time to DateTime, the missing Date part defaults to December 30th, 1899:

One solution is to replace the Activity[Time] column reference in these calculated column formulas with Activity[Date] & " " & Activity[Time]:

LastUsageDateTimeDataset = 
    var v_id = 'Catalog - Dataset'[DatasetId]
    return MAXX(
        FILTER('Activity', Activity[DatasetId] = v_id), 
        Activity[Date] & " " & Activity[Time]
    )

LastUsageDateTimeReport = 
    var v_id = 'Catalog - Report'[ReportId]
    return MAXX(
        FILTER('Activity', Activity[ReportId] = v_id), 
        Activity[Date] & " " & Activity[Time]
    )

I wanted to open the floor for discussion about the best way to solve this problem, and any potential issues that might arise from implementing it the way I've proposed above. If I haven't heard any disagreement with my proposed solution in a few weeks time, I'll make the change myself and submit a PR.

Not all datasets return schema detail

Hi Rui,

Fantastic tool and repo...thank you! We are running into an issue that we have been unable to resolve.

Some but not all datasets do not return any schema info. In particular, our main enterprise data model is present in each of our three environments (Dev, Test, Prod). It is promoted through environments via a deployment pipeline. Pbimonitor returns schema information for the dataset in Dev and Test but does not return it for the dataset in Prod. All the correct tenant settings have been enabled.

For troubleshooting, we deleted all of the subfolders in the "Data" folder along with the state file to force a full scan. We also set the catalog URI parameters to:

GetModified parameters 'excludePersonalWorkspaces=false&excludeInActiveWorkspaces=false'
GetInfo parameters 'lineage=true&datasourceDetails=true&getArtifactUsers=true&datasetSchema=true&datasetExpressions=true'

Do you have any suggestions? Are there any settings/characteristics of a dataset that would prevent it from returning schema data?

It is probably unrelated but the "Activity" script has also thrown the following error which was resolved by modifying the depth parameter to 6:

Getting audit data for: '20221005'
Writing '1844' audits
WARNING: Resulting JSON is truncated as serialization has exceeded the set depth of 5.

Optimize 'Permissions' query

No data for Tables/Columns

Hi, I'm using your Azure Function and I noticed that the default value for "$getInfoDetails" in "Fetch - Catalog.ps1" is:

$getInfoDetails = "lineage=true&datasourceDetails=true&getArtifactUsers=true&datasetSchema=false&datasetExpressions=false"

"datasetSchema=false" and "datasetExpressions=false" is excluding tables/columns details right?

I don't have the PBIMONITOR_CatalogGetInfoParameters Application Setting in the Azure Function Configuration, would that be the reason? If so, what value should I use? "lineage=true&datasourceDetails=true&getArtifactUsers=true&datasetSchema=TRUE&datasetExpressions=TRUE"?

Thanks for your solution btw, It's been really helpful!!!
Vanessa

AuditsTimer 403 (Forbidden)

I went to test the AuditsTimer function within the function app and received 403 (Forbidden) error messages (see excerpt below). Not super savvy with Azure, any idea what this could be? Something to do with the created secret or privileges of my Azure account?

2024-04-15T07:37:36Z   [Information]   Executing 'Functions.AuditsTimer' (Reason='This function was programmatically called via the host APIs.', Id=7d871506-8bce-4c12-9804-8f2755a3193a)
2024-04-15T07:37:36Z   [Verbose]   Sending invocation id: '7d871506-8bce-4c12-9804-8f2755a3193a
2024-04-15T07:37:36Z   [Verbose]   Posting invocation id:7d871506-8bce-4c12-9804-8f2755a3193a on workerId:49abab0d-03a8-40d9-a47c-02c63f200c54
2024-04-15T07:37:46Z   [Information]   INFORMATION: PBIMonitor - Fetch Activity Started: 04/15/2024 07:37:45
2024-04-15T07:37:46Z   [Information]   INFORMATION: Building PBIMonitor Config from Azure Function Configuration
2024-04-15T07:37:46Z   [Information]   INFORMATION: AppDataPath: C:\home\data\pbimonitor
2024-04-15T07:37:46Z   [Information]   INFORMATION: ScriptsPath: C:\home\site\wwwroot\Scripts
2024-04-15T07:37:46Z   [Information]   INFORMATION: OutputPath: C:\local\Temp\PBIMonitorData\0f2337d7e3774480b53b8c140c0c7327
2024-04-15T07:37:47Z   [Information]   INFORMATION: Starting Power BI Activity Fetch
2024-04-15T07:37:47Z   [Information]   INFORMATION: Since: 2024-03-16T00:00:00
2024-04-15T07:37:47Z   [Information]   INFORMATION: OutputBatchCount: 5000
2024-04-15T07:37:47Z   [Information]   INFORMATION: Getting OAuth Token
2024-04-15T07:37:48Z   [Information]   INFORMATION: Login with: e82caa08-8c62-4d4c-8c7c-3a8207c3c4b2
2024-04-15T07:37:48Z   [Information]   INFORMATION: Getting audit data for: '20240316'
2024-04-15T07:37:48Z   [Information]   INFORMATION: Ellapsed: 1.3668811s
2024-04-15T07:37:48Z   [Information]   OUTPUT: 
2024-04-15T07:37:48Z   [Information]   OUTPUT:    HistoryId: 1
2024-04-15T07:37:48Z   [Information]   OUTPUT: 
2024-04-15T07:37:48Z   [Information]   OUTPUT: Message        : Response status code does not indicate success: 403 (Forbidden).
2024-04-15T07:37:48Z   [Information]   OUTPUT: StackTrace     :    at Microsoft.PowerBI.Commands.Profile.InvokePowerBIRestMethod.InvokeRestMethod(String url, String body, PowerBIWebRequestMethod requestType)
2024-04-15T07:37:48Z   [Information]   OUTPUT: Exception      : System.Net.Http.HttpRequestException
2024-04-15T07:37:48Z   [Information]   OUTPUT: InvocationInfo : {Invoke-PowerBIRestMethod}
2024-04-15T07:37:48Z   [Information]   OUTPUT: Line           :                 $result = Invoke-PowerBIRestMethod -Url $activityAPIUrl -method Get | ConvertFrom-Json
2024-04-15T07:37:48Z   [Information]   OUTPUT:                  
2024-04-15T07:37:48Z   [Information]   OUTPUT: Position       : At C:\home\site\wwwroot\Scripts\Fetch - Activity.ps1:90 char:27
2024-04-15T07:37:48Z   [Information]   OUTPUT:                  + …   $result = Invoke-PowerBIRestMethod -Url $activityAPIUrl -method Get …
2024-04-15T07:37:48Z   [Information]   OUTPUT:                  +               ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2024-04-15T07:37:48Z   [Information]   OUTPUT: HistoryId      : 1
2024-04-15T07:37:48Z   [Information]   OUTPUT: 
2024-04-15T07:37:48Z   [Information]   OUTPUT: Message        : Response status code does not indicate success: 403 (Forbidden).
2024-04-15T07:37:48Z   [Information]   OUTPUT: StackTrace     :    at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
2024-04-15T07:37:48Z   [Information]   OUTPUT:                     at Microsoft.PowerBI.Commands.Profile.InvokePowerBIRestMethod.InvokeRestMethod(String url, String body, PowerBIWebRequestMethod requestType)
2024-04-15T07:37:48Z   [Information]   OUTPUT: Exception      : System.Net.Http.HttpRequestException
2024-04-15T07:37:48Z   [Information]   OUTPUT: InvocationInfo : {Invoke-PowerBIRestMethod}
2024-04-15T07:37:48Z   [Information]   OUTPUT: Line           :                 $result = Invoke-PowerBIRestMethod -Url $activityAPIUrl -method Get | ConvertFrom-Json
2024-04-15T07:37:48Z   [Information]   OUTPUT:                  
2024-04-15T07:37:48Z   [Information]   OUTPUT: Position       : At C:\home\site\wwwroot\Scripts\Fetch - Activity.ps1:90 char:27
2024-04-15T07:37:48Z   [Information]   OUTPUT:                  + …   $result = Invoke-PowerBIRestMethod -Url $activityAPIUrl -method Get …
2024-04-15T07:37:48Z   [Information]   OUTPUT:                  +               ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2024-04-15T07:37:48Z   [Information]   OUTPUT: HistoryId      : 1
2024-04-15T07:37:48Z   [Information]   OUTPUT: 
2024-04-15T07:37:49Z   [Error]   EXCEPTION: One or more errors occurred. (Response status code does not indicate success: 403 (Forbidden).)

Exception             : 
    Type            : System.AggregateException
    InnerExceptions : 
        Type           : System.Net.Http.HttpRequestException
        TargetSite     : 
            Name          : MoveNext
            DeclaringType : Microsoft.PowerBI.Commands.Profile.InvokePowerBIRestMethod+<InvokeRestMethod>d__35, Microsoft.PowerBI.Commands.Profile, Version=1.2.1111.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35
            MemberType    : Method
            Module        : Microsoft.PowerBI.Commands.Profile.dll
        Message        : Response status code does not indicate success: 403 (Forbidden).
        InnerException : 
            Type       : System.Net.Http.HttpRequestException
            StatusCode : Forbidden
            TargetSite : 
                Name          : EnsureSuccessStatusCode
                DeclaringType : System.Net.Http.HttpResponseMessage
                MemberType    : Method
                Module        : System.Net.Http.dll
            Message    : Response status code does not indicate success: 403 (Forbidden).
            Source     : System.Net.Http
            HResult    : -2146233088
            StackTrace : 
   at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
   at Microsoft.PowerBI.Commands.Profile.InvokePowerBIRestMethod.InvokeRestMethod(String url, String body, PowerBIWebRequestMethod requestType)
        Source         : Microsoft.PowerBI.Commands.Profile
        HResult        : -2146233088
        StackTrace     : 
   at Microsoft.PowerBI.Commands.Profile.InvokePowerBIRestMethod.InvokeRestMethod(String url, String body, PowerBIWebRequestMethod requestType)
    Message         : One or more errors occurred. (Response status code does not indicate success: 403 (Forbidden).)
    TargetSite      : 
        Name          : ThrowIfExceptional
        DeclaringType : System.Threading.Tasks.Task
        MemberType    : Method
        Module        : System.Private.CoreLib.dll
    InnerException  : 
        Type           : System.Net.Http.HttpRequestException
        TargetSite     : 
            Name          : MoveNext
            DeclaringType : Microsoft.PowerBI.Commands.Profile.InvokePowerBIRestMethod+<InvokeRestMethod>d__35, Microsoft.PowerBI.Commands.Profile, Version=1.2.1111.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35
            MemberType    : Method
            Module        : Microsoft.PowerBI.Commands.Profile.dll
        Message        : Response status code does not indicate success: 403 (Forbidden).
        InnerException : 
            Type       : System.Net.Http.HttpRequestException
            StatusCode : Forbidden
            TargetSite : 
                Name          : EnsureSuccessStatusCode
                DeclaringType : System.Net.Http.HttpResponseMessage
                MemberType    : Method
                Module        : System.Net.Http.dll
            Message    : Response status code does not indicate success: 403 (Forbidden).
            Source     : System.Net.Http
            HResult    : -2146233088
            StackTrace : 
   at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
   at Microsoft.PowerBI.Commands.Profile.InvokePowerBIRestMethod.InvokeRestMethod(String url, String body, PowerBIWebRequestMethod requestType)
        Source         : Microsoft.PowerBI.Commands.Profile
        HResult        : -2146233088
        StackTrace     : 
   at Microsoft.PowerBI.Commands.Profile.InvokePowerBIRestMethod.InvokeRestMethod(String url, String body, PowerBIWebRequestMethod requestType)
    Source          : System.Private.CoreLib
    HResult         : -2146233088
    StackTrace      : 
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at System.Threading.Tasks.Task`1.get_Result()
   at Microsoft.PowerBI.Commands.Profile.InvokePowerBIRestMethod.ExecuteCmdlet()
   at Microsoft.PowerBI.Commands.Common.PowerBICmdlet.ProcessRecord()
TargetObject          : Microsoft.PowerBI.Commands.Profile.InvokePowerBIRestMethod
CategoryInfo          : WriteError: (Microsoft.PowerBI.C…kePowerBIRestMethod:InvokePowerBIRestMethod) [Invoke-PowerBIRestMethod], AggregateException
FullyQualifiedErrorId : One or more errors occurred. (Response status code does not indicate success: 403 (Forbidden).),Microsoft.PowerBI.Commands.Profile.InvokePowerBIRestMethod
InvocationInfo        : 
    MyCommand        : Invoke-PowerBIRestMethod
    ScriptLineNumber : 90
    OffsetInLine     : 27
    HistoryId        : 1
    ScriptName       : C:\home\site\wwwroot\Scripts\Fetch - Activity.ps1
    Line             : $result = Invoke-PowerBIRestMethod -Url $activityAPIUrl -method Get | ConvertFrom-Json
                       
    Statement        : Invoke-PowerBIRestMethod -Url $activityAPIUrl -method Get
    PositionMessage  : At C:\home\site\wwwroot\Scripts\Fetch - Activity.ps1:90 char:27
                       + …   $result = Invoke-PowerBIRestMethod -Url $activityAPIUrl -method Get …
                       +               ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    PSScriptRoot     : C:\home\site\wwwroot\Scripts
    PSCommandPath    : C:\home\site\wwwroot\Scripts\Fetch - Activity.ps1
    InvocationName   : Invoke-PowerBIRestMethod
    CommandOrigin    : Internal
ScriptStackTrace      : at <ScriptBlock>, C:\home\site\wwwroot\Scripts\Fetch - Activity.ps1: line 90
                        at <ScriptBlock>, C:\home\site\wwwroot\AuditsTimer\run.ps1: line 31
PipelineIterationInfo : 







2024-04-15T07:37:49Z   [Error]   Executed 'Functions.AuditsTimer' (Failed, Id=7d871506-8bce-4c12-9804-8f2755a3193a, Duration=12467ms)

Support other regions in the config file

FetchGraph.ps1

Hi,

This uses access token. Is there another way to login interactively and extract this data?

Thanks.

System.AggregateException

Running the PowerShell script "Fetch - Run.ps1" as normal with default full refresh cadence of 30days returned the below error after serveral days. The issue appered during the catalog part.
I'm wondering, if there is a certain file size limit for the catalog json files. The file size of the full refresh json files is ~30-70MB, while for the incremental refreshed I get up to 200MB.

'Alerts' Page

Tools for PBI admins to find things that need fixing

429 issue for data catalog in large tenant

When executing a full run of the scanner API (catalog.ps1) without the personal workspaces the script is running perfectly fine. However, adding the personal workspaces gets us to ~70k workspaces and therefore over the throttling limit of 500 (x 100 workspaces) requests per hour. I'm not any expert with PowerShell, but I was assuming that the script is taking care of that with the line 239 in script catalog.ps1:

Wait-On429Error -tentatives 1 -sleepSeconds $throttleErrorSleepSeconds -script {

However, the script is stopping with the known 429 error message:

C:\PBIMonitor\Fetch - Run.ps1 : Error on 'C:\PBIMonitor\Fetch - Catalog.ps1' - System.AggregateException: Mindestens ein Fehler ist aufgetreten. ---> System.Net.Http.HttpRequestException: Der Antwortstatuscode gibt
keinen Erfolg an: 429 ().
bei System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
bei Microsoft.PowerBI.Commands.Profile.InvokePowerBIRestMethod.d__35.MoveNext()
--- Ende der internen Ausnahmestapelüberwachung ---
bei System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
bei Microsoft.PowerBI.Commands.Profile.InvokePowerBIRestMethod.ExecuteCmdlet()
bei Microsoft.PowerBI.Commands.Common.PowerBICmdlet.ProcessRecord()
---> (Interne Ausnahme #0) System.Net.Http.HttpRequestException: Der Antwortstatuscode gibt keinen Erfolg an: 429 ().
bei System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
bei Microsoft.PowerBI.Commands.Profile.InvokePowerBIRestMethod.d__35.MoveNext()<---
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Fetch - Run.ps1

Should the script take care of the 500 request/hour limit? Why is it not working correctly?
If not, how can we get around it?

Give access to the function managed identity to write in the storage account

Amazing piece of work!
just wanted to mention something I experienced

I think that this step is not mentioned (might have missed it)
but you need to give permision to the funciton to write on the storage account. no big deal, but necessary for the functions to work

CatalogTImer Function Issue

Hi,

We are facing issue while using the CatalogTimer function, which was working previously, but after changing the secret key we are getting errors. Please find the below error captured from Minitor log.

EXCEPTION: One or more errors occurred. (Error reading JObject from JsonReader. Current JsonReader item is not an object: String. Path '', line 1, position 131.) Exception : Type : System.AggregateException InnerExceptions : Type : Newtonsoft.Json.JsonReaderException LineNumber : 1 LinePosition : 131 TargetSite : Name : Load DeclaringType : Newtonsoft.Json.Linq.JObject MemberType : Method Module : Newtonsoft.Json.dll Message : Error reading JObject from JsonReader. Current JsonReader item is not an object: String. Path '', line 1, position 131. Source : Newtonsoft.Json HResult : -2146233088 StackTrace : at Newtonsoft.Json.Linq.JObject.Load(JsonReader reader, JsonLoadSettings settings) at Newtonsoft.Json.Linq.JObject.Parse(String json, JsonLoadSettings settings) at Newtonsoft.Json.Linq.JObject.Parse(String json) at Microsoft.PowerBI.Commands.Profile.InvokePowerBIRestMethod.InvokeRestMethod(String url, String body, PowerBIWebRequestMethod requestType) Message : One or more errors occurred. (Error reading JObject from JsonReader. Current JsonReader item is not an object: String. Path '', line 1, position 131.) TargetSite : Name : ThrowIfExceptional DeclaringType : System.Threading.Tasks.Task MemberType : Method Module : System.Private.CoreLib.dll InnerException : Type : Newtonsoft.Json.JsonReaderException LineNumber : 1 LinePosition : 131 TargetSite : Name : Load DeclaringType : Newtonsoft.Json.Linq.JObject MemberType : Method Module : Newtonsoft.Json.dll Message : Error reading JObject from JsonReader. Current JsonReader item is not an object: String. Path '', line 1, position 131. Source : Newtonsoft.Json HResult : -2146233088 StackTrace : at Newtonsoft.Json.Linq.JObject.Load(JsonReader reader, JsonLoadSettings settings) at Newtonsoft.Json.Linq.JObject.Parse(String json, JsonLoadSettings settings) at Newtonsoft.Json.Linq.JObject.Parse(String json) at Microsoft.PowerBI.Commands.Profile.InvokePowerBIRestMethod.InvokeRestMethod(String url, String body, PowerBIWebRequestMethod requestType) Source : System.Private.CoreLib HResult : -2146233088 StackTrace : at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions) at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification) at Microsoft.PowerBI.Commands.Profile.InvokePowerBIRestMethod.ExecuteCmdlet() at Microsoft.PowerBI.Commands.Common.PowerBICmdlet.ProcessRecord() TargetObject : Microsoft.PowerBI.Commands.Profile.InvokePowerBIRestMethod CategoryInfo : WriteError: (Microsoft.PowerBI.C…kePowerBIRestMethod:InvokePowerBIRestMethod) [Invoke-PowerBIRestMethod], AggregateException FullyQualifiedErrorId : One or more errors occurred. (Error reading JObject from JsonReader. Current JsonReader item is not an object: String. Path '', line 1, position 131.),Microsoft.PowerBI.Commands.Profile.InvokePowerBIRestMethod InvocationInfo : MyCommand : Invoke-PowerBIRestMethod ScriptLineNumber : 213 OffsetInLine : 40 HistoryId : 1 ScriptName : C:\home\site\wwwroot\Scripts\Fetch - Catalog.ps1 Line : $getInfoResult = @(Invoke-PowerBIRestMethod -Url "admin/workspaces/getInfo?$getInfoDetails" -Body $bodyStr -method Post | ConvertFrom-Json) PositionMessage : At C:\home\site\wwwroot\Scripts\Fetch - Catalog.ps1:213 char:40 + … oResult = @(Invoke-PowerBIRestMethod -Url "admin/workspaces/getInfo?$ … + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ PSScriptRoot : C:\home\site\wwwroot\Scripts PSCommandPath : C:\home\site\wwwroot\Scripts\Fetch - Catalog.ps1 InvocationName : Invoke-PowerBIRestMethod CommandOrigin : Internal ScriptStackTrace : at , C:\home\site\wwwroot\Scripts\Fetch - Catalog.ps1: line 213 at Wait-On429Error, C:\home\site\wwwroot\Scripts\Fetch - Utils.psm1: line 184 at , C:\home\site\wwwroot\Scripts\Fetch - Catalog.ps1: line 207 at Get-ArrayInBatches, C:\home\site\wwwroot\Scripts\Fetch - Utils.psm1: line 159 at , C:\home\site\wwwroot\Scripts\Fetch - Catalog.ps1: line 204 at Get-ArrayInBatches, C:\home\site\wwwroot\Scripts\Fetch - Utils.psm1: line 159 at , C:\home\site\wwwroot\Scripts\Fetch - Catalog.ps1: line 198 at , C:\home\site\wwwroot\CatalogTimer\run.ps1: line 31 PipelineIterationInfo :

Showing GUI for the Power BI Service Principals

Hello Ruy

It seems the service principals is showing the GUI not the User name.

Power BI Monitoring - issues with Graph API

Hi RuiRomano,

I've implemented the solution and managed to get most of it working, however I'm have issue retrieving Licence and User data from the Graph API.

Therefore the Licence and User tab has no data within the Power BI report.

Please could you help with this issue.. I believe it might be some kind of authorisation, as the rest of the report works

Any guidance will be much appreciated.

Token not refreshing if script is running for more than an hour

I am having an issue with the token expiring if the script is running for more than an hour. The script is processing entries fine then after an hour we start getting "Your access token has expired". See error message from transcript below.

TerminatingError(Invoke-RestMethod): "{"error":{"code":"Authentication_ExpiredToken","message":"Your access token has expired. Please renew it before submitting the request.","innerError":{"date":"2023-07-26T17:16:42","request-id":"","client-request-id":""}}}"

Change PBIX to bring dataset tables, columns, measures, DAX expressions, and mashup queries into the report

It would be good to do this both for PowerShell and Azure Functions approach

Out-File keeps failing when running Fetch-Activity.ps1

When I run the single file Fetch-Activity.ps1 I keep getting the following error:

Seems like the access to path, or the path formation in line 155 is incorrect or not fully building the output path correctly.

If I only run Fetch-Activity.ps1 from the Fetch-Run.ps1 file I get a different error:

Timers not running

the following happens when invoking any of the timers:

2023-06-29T10:46:17Z   [Information]   Executing 'Functions.AuditsTimer' (Reason='This function was programmatically called via the host APIs.', Id=2c6369ab-d8a9-4746-9212-381d303d65e1)
2023-06-29T10:46:17Z   [Verbose]   Sending invocation id: '2c6369ab-d8a9-4746-9212-381d303d65e1
2023-06-29T10:46:17Z   [Verbose]   Posting invocation id:2c6369ab-d8a9-4746-9212-381d303d65e1 on workerId:b1ee0548-2130-46aa-94d3-d8e48d96a92a
2023-06-29T10:46:18Z   [Information]   INFORMATION: PBIMonitor - Fetch Activity Started: 06/29/2023 10:46:17
2023-06-29T10:46:18Z   [Information]   INFORMATION: Building PBIMonitor Config from Azure Function Configuration
2023-06-29T10:46:18Z   [Information]   INFORMATION: AppDataPath: C:\home\data\pbimonitor
2023-06-29T10:46:18Z   [Information]   INFORMATION: ScriptsPath: C:\home\site\wwwroot\Scripts
2023-06-29T10:46:18Z   [Information]   INFORMATION: OutputPath: C:\local\Temp\PBIMonitorData\87508303bdbb40dba1b3934e70d00585
2023-06-29T10:46:18Z   [Information]   INFORMATION: Starting Power BI Activity Fetch
2023-06-29T10:46:18Z   [Information]   INFORMATION: Since: 2023-05-30T00:00:00
2023-06-29T10:46:18Z   [Information]   INFORMATION: OutputBatchCount: 5000
2023-06-29T10:46:18Z   [Information]   INFORMATION: Ellapsed: 0.005944s
2023-06-29T10:46:18Z   [Information]   INFORMATION: Getting OAuth Token
2023-06-29T10:46:18Z   [Information]   OUTPUT: 
2023-06-29T10:46:18Z   [Information]   OUTPUT:    HistoryId: 1
2023-06-29T10:46:18Z   [Information]   OUTPUT: 
2023-06-29T10:46:18Z   [Information]   OUTPUT: Message        : Cannot bind argument to parameter 'String' because it is an empty string.
2023-06-29T10:46:18Z   [Information]   OUTPUT: StackTrace     :    at System.Management.Automation.CmdletParameterBinderController.BindValueFromPipeline(PSObject inputToOperateOn, MergedCompiledCommandParameter parameter, ParameterBindingFlags flags)
2023-06-29T10:46:18Z   [Information]   OUTPUT:                     at System.Management.Automation.CmdletParameterBinderController.BindUnboundParametersForBindingStateInParameterSet(PSObject inputToOperateOn, CurrentlyBinding currentlyBinding, UInt32 validParameterSets)
2023-06-29T10:46:18Z   [Information]   OUTPUT:                     at System.Management.Automation.CmdletParameterBinderController.BindUnboundParametersForBindingState(PSObject inputToOperateOn, CurrentlyBinding currentlyBinding, UInt32 validParameterSets)
2023-06-29T10:46:18Z   [Information]   OUTPUT:                     at System.Management.Automation.CmdletParameterBinderController.BindPipelineParametersPrivate(PSObject inputToOperateOn)
2023-06-29T10:46:18Z   [Information]   OUTPUT:                     at System.Management.Automation.CmdletParameterBinderController.BindPipelineParameters(PSObject inputToOperateOn)
2023-06-29T10:46:18Z   [Information]   OUTPUT:                     at System.Management.Automation.CommandProcessor.ProcessInputPipelineObject(Object inputObject)
2023-06-29T10:46:18Z   [Information]   OUTPUT:                     at System.Management.Automation.CommandProcessor.Read()
2023-06-29T10:46:18Z   [Information]   OUTPUT: Exception      : System.Management.Automation.ParameterBindingValidationException
2023-06-29T10:46:18Z   [Information]   OUTPUT: InvocationInfo : {ConvertTo-SecureString}
2023-06-29T10:46:18Z   [Information]   OUTPUT: Line           :         $credential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $config.ServicePrincipal.AppId, ($config.ServicePrincipal.AppSecret | ConvertTo-SecureString -AsPlainText -Force)
2023-06-29T10:46:18Z   [Information]   OUTPUT:                  
2023-06-29T10:46:18Z   [Information]   OUTPUT: Position       : At C:\home\site\wwwroot\Scripts\Fetch - Activity.ps1:65 char:170
2023-06-29T10:46:18Z   [Information]   OUTPUT:                  + … vicePrincipal.AppSecret | ConvertTo-SecureString -AsPlainText -Force)
2023-06-29T10:46:18Z   [Information]   OUTPUT:                  +                             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2023-06-29T10:46:18Z   [Information]   OUTPUT: HistoryId      : 1
2023-06-29T10:46:18Z   [Information]   OUTPUT: 
2023-06-29T10:46:18Z   [Error]   EXCEPTION: Cannot bind argument to parameter 'String' because it is an empty string.

Exception             : 
    Type              : System.Management.Automation.ParameterBindingValidationException
    Message           : Cannot bind argument to parameter 'String' because it is an empty string.
    ParameterName     : String
    ParameterType     : string
    TypeSpecified     : string
    ErrorId           : ParameterArgumentValidationErrorEmptyStringNotAllowed
    Line              : 65
    Offset            : 170
    CommandInvocation : 
        MyCommand        : ConvertTo-SecureString
        BoundParameters  : 
            Comparer : System.OrdinalIgnoreCaseComparer
            Count    : 2
            Keys     : 
                Length : 11

                Length : 5
            Values   : 
                IsPresent : True

                IsPresent : True
            SyncRoot : 
                Comparer : System.OrdinalIgnoreCaseComparer
                Count    : 2
                Keys     : 
                    Length : 11

                    Length : 5
                Values   : 
                    IsPresent : True

                    IsPresent : True
                SyncRoot : 
                    Comparer : System.OrdinalIgnoreCaseComparer
                    Count    : 2
                    Keys     : 
                        Length : 11

                        Length : 5
                    Values   : 
                        IsPresent : True

                        IsPresent : True
                    SyncRoot : 
                        Comparer : System.OrdinalIgnoreCaseComparer
                        Count    : 2
                        Keys     : 
                            Length : 11

                            Length : 5
                        Values   : 
                            IsPresent : True

                            IsPresent : True
                        SyncRoot : 
                            Comparer : System.OrdinalIgnoreCaseComparer
                            Count    : 2
                            Keys     : 
                                Length : 11

                                Length : 5
                            Values   : 
                                IsPresent : True

                                IsPresent : True
                            SyncRoot : 
                                Comparer : System.OrdinalIgnoreCaseComparer
                                Count    : 2
                                Keys     : 
                                    Length : 11

                                    Length : 5
                                Values   : 
                                    IsPresent : True

                                    IsPresent : True
                                SyncRoot : 
                                    Comparer : System.OrdinalIgnoreCaseComparer
                                    Count    : 2
                                    Keys     : …
                                    Values   : …
                                    SyncRoot : …
        ScriptLineNumber : 65
        OffsetInLine     : 170
        HistoryId        : 1
        ScriptName       : C:\home\site\wwwroot\Scripts\Fetch - Activity.ps1
        Line             : $credential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $config.ServicePrincipal.AppId, ($config.ServicePrincipal.AppSecret | ConvertTo-SecureString -AsPlainText -Force)
                           
        PositionMessage  : At C:\home\site\wwwroot\Scripts\Fetch - Activity.ps1:65 char:170
                           + … vicePrincipal.AppSecret | ConvertTo-SecureString -AsPlainText -Force)
                           +                             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        PSScriptRoot     : C:\home\site\wwwroot\Scripts
        PSCommandPath    : C:\home\site\wwwroot\Scripts\Fetch - Activity.ps1
        InvocationName   : ConvertTo-SecureString
        PipelineLength   : 1
        PipelinePosition : 1
        ExpectingInput   : True
        CommandOrigin    : Internal
    ErrorRecord       : 
        Exception             : 
            Type    : System.Management.Automation.ParentContainsErrorRecordException
            Message : Cannot bind argument to parameter 'String' because it is an empty string.
            HResult : -2146233087
        CategoryInfo          : InvalidData: (:String) [ConvertTo-SecureString], ParentContainsErrorRecordException
        FullyQualifiedErrorId : ParameterArgumentValidationErrorEmptyStringNotAllowed,Microsoft.PowerShell.Commands.ConvertToSecureStringCommand
        InvocationInfo        : 
            MyCommand        : ConvertTo-SecureString
            ScriptLineNumber : 65
            OffsetInLine     : 170
            HistoryId        : 1
            ScriptName       : C:\home\site\wwwroot\Scripts\Fetch - Activity.ps1
            Line             : $credential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $config.ServicePrincipal.AppId, ($config.ServicePrincipal.AppSecret | ConvertTo-SecureString -AsPlainText -Force)
                               
            PositionMessage  : At C:\home\site\wwwroot\Scripts\Fetch - Activity.ps1:65 char:170
                               + … vicePrincipal.AppSecret | ConvertTo-SecureString -AsPlainText -Force)
                               +                             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            PSScriptRoot     : C:\home\site\wwwroot\Scripts
            PSCommandPath    : C:\home\site\wwwroot\Scripts\Fetch - Activity.ps1
            CommandOrigin    : Internal
        ScriptStackTrace      : at <ScriptBlock>, C:\home\site\wwwroot\Scripts\Fetch - Activity.ps1: line 65
                                at <ScriptBlock>, C:\home\site\wwwroot\AuditsTimer\run.ps1: line 31
    TargetSite        : 
        Name          : BindValueFromPipeline
        DeclaringType : System.Management.Automation.CmdletParameterBinderController, System.Management.Automation, Version=7.2.11.500, Culture=neutral, PublicKeyToken=31bf3856ad364e35
        MemberType    : Method
        Module        : System.Management.Automation.dll
    Source            : System.Management.Automation
    HResult           : -2146233087
    StackTrace        : 
   at System.Management.Automation.CmdletParameterBinderController.BindValueFromPipeline(PSObject inputToOperateOn, MergedCompiledCommandParameter parameter, ParameterBindingFlags flags)
   at System.Management.Automation.CmdletParameterBinderController.BindUnboundParametersForBindingStateInParameterSet(PSObject inputToOperateOn, CurrentlyBinding currentlyBinding, UInt32 validParameterSets)
   at System.Management.Automation.CmdletParameterBinderController.BindUnboundParametersForBindingState(PSObject inputToOperateOn, CurrentlyBinding currentlyBinding, UInt32 validParameterSets)
   at System.Management.Automation.CmdletParameterBinderController.BindPipelineParametersPrivate(PSObject inputToOperateOn)
   at System.Management.Automation.CmdletParameterBinderController.BindPipelineParameters(PSObject inputToOperateOn)
   at System.Management.Automation.CommandProcessor.ProcessInputPipelineObject(Object inputObject)
   at System.Management.Automation.CommandProcessor.Read()
CategoryInfo          : InvalidData: (:String) [ConvertTo-SecureString], ParameterBindingValidationException
FullyQualifiedErrorId : ParameterArgumentValidationErrorEmptyStringNotAllowed,Microsoft.PowerShell.Commands.ConvertToSecureStringCommand
InvocationInfo        : 
    MyCommand        : ConvertTo-SecureString
    ScriptLineNumber : 65
    OffsetInLine     : 170
    HistoryId        : 1
    ScriptName       : C:\home\site\wwwroot\Scripts\Fetch - Activity.ps1
    Line             : $credential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $config.ServicePrincipal.AppId, ($config.ServicePrincipal.AppSecret | ConvertTo-SecureString -AsPlainText -Force)
                       
    PositionMessage  : At C:\home\site\wwwroot\Scripts\Fetch - Activity.ps1:65 char:170
                       + … vicePrincipal.AppSecret | ConvertTo-SecureString -AsPlainText -Force)
                       +                             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    PSScriptRoot     : C:\home\site\wwwroot\Scripts
    PSCommandPath    : C:\home\site\wwwroot\Scripts\Fetch - Activity.ps1
    CommandOrigin    : Internal
ScriptStackTrace      : at <ScriptBlock>, C:\home\site\wwwroot\Scripts\Fetch - Activity.ps1: line 65
                        at <ScriptBlock>, C:\home\site\wwwroot\AuditsTimer\run.ps1: line 31


2023-06-29T10:46:18Z   [Error]   Executed 'Functions.AuditsTimer' (Failed, Id=2c6369ab-d8a9-4746-9212-381d303d65e1, Duration=335ms)

Any help greatly appreciated!

Support Security Groups and map to users

GraphTimer Error 403 Forbidden

Hello,
It AuditTimer and CatagTimer are working fine for me but i am getting Error 403 Forbidden when i run Graphtimer, any idea ?

2021-12-23T10:34:48.607 [Information] INFORMATION: Ellapsed: 0.2177514s
2021-12-23T10:34:48.922 [Information] OUTPUT:
2021-12-23T10:34:48.923 [Information] OUTPUT:
2021-12-23T10:34:48.923 [Information] OUTPUT: HistoryId: 1
2021-12-23T10:34:48.923 [Information] OUTPUT:
2021-12-23T10:34:48.923 [Information] OUTPUT: Message : Response status code does not indicate success: 403 (Forbidden).
2021-12-23T10:34:48.923 [Information] OUTPUT: StackTrace : at System.Management.Automation.MshCommandRuntime.ThrowTerminatingError(ErrorRecord errorRecord)
2021-12-23T10:34:48.924 [Information] OUTPUT: Exception : Microsoft.PowerShell.Commands.HttpResponseException
2021-12-23T10:34:48.924 [Information] OUTPUT: InvocationInfo : {Invoke-RestMethod}
2021-12-23T10:34:48.924 [Information] OUTPUT: Line : $result = Invoke-RestMethod -Method Get -Uri $url -Headers $headers
2021-12-23T10:34:48.924 [Information] OUTPUT:
2021-12-23T10:34:48.924 [Information] OUTPUT: Position : At C:\home\site\wwwroot\Scripts\Fetch - Graph.ps1:61 char:19
2021-12-23T10:34:48.924 [Information] OUTPUT: + … $result = Invoke-RestMethod -Method Get -Uri $url -Headers $headers …
2021-12-23T10:34:48.924 [Information] OUTPUT: + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-12-23T10:34:48.924 [Information] OUTPUT: HistoryId : 1
2021-12-23T10:34:48.924 [Information] OUTPUT:
2021-12-23T10:34:49.366 [Error] EXCEPTION: Response status code does not indicate success: 403 (Forbidden).Exception :Type : Microsoft.PowerShell.Commands.HttpResponseExceptionResponse : StatusCode: 403, ReasonPhrase: 'Forbidden', Version: 1.1, Content: System.Net.Http.HttpConnectionResponseContent, Headers:{Date: Thu, 23 Dec 2021 10:34:48 GMTCache-Control: no-cacheTransfer-Encoding: chunkedStrict-Transport-Security: max-age=31536000request-id: 7731876f-d05c-474c-b082-57a7efdd63f3client-request-id: 7731876f-d05c-474c-b082-57a7efdd63f3x-ms-ags-diagnostic: {"ServerInfo":{"DataCenter":"North Europe","Slice":"E","Ring":"4","ScaleUnit":"005","RoleInstance":"DU6PEPF00001536"}}x-ms-resource-unit: 1Content-Type: application/json}TargetSite :Name : ThrowTerminatingErrorDeclaringType : System.Management.Automation.MshCommandRuntime, System.Management.Automation, Version=7.0.7.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35MemberType : MethodModule : System.Management.Automation.dllStackTrace :at System.Management.Automation.MshCommandRuntime.ThrowTerminatingError(ErrorRecord errorRecord)Message : Response status code does not indicate success: 403 (Forbidden).Source : System.Management.AutomationHResult : -2146233088TargetObject : Method: GET, RequestUri: 'https://graph.microsoft.com/beta/users?$select=id,mail,companyName,displayName,assignedLicenses,onPremisesUserPrincipalName,UserPrincipalName,jobTitle,userType', Version: 1.1, Content: System.Net.Http.StringContent,

FetchActivity

when i run your FetchActivity script, for some reason i will not return the log.

The strange thing is, i seems as it knows how many logs there should be (there is a differents in how many objects it retrives but they are all empty)

Port to Fabric notebooks

Hey Rui,

I'm wondering if you are planning to port this amazing solution to Microsoft Fabric using

Notebooks
Pipelines
Lakehouse
Direct Lake dataset

All the best
Tom

Add domain workspace support

Hello Rui,
PBI Monitor is a great solution !!!
There is a new setting on Worksplace since few month: Domain.
It could be great to integrate this setting within the data you retrieve from PowerBI workspaces informations and then update the report to be able to filter it on Domain so that can provide access to this report to domain owners and they will see only information linked to their scope.
It can be also used for governance setup by identifying all workspace without domain setup.

Error 429 - Throtteling

Hello,
I work for a company with 120K workspaces, including personal spaces.
I can only recover 50K workspaces and then I get error 429 - Throtteling.

I switched to Plan premium on azure functions but it didn't help.

Do you have a patch to get the WS from the whole tenant power bi?

Thank you very much.

mention operating system in hosting

I made a mistake in choosing linux as my os. I didn't remember that linux azure function does not have the advance settings kudu. I can map the paths though to these below

Windows Path: C:\home\data\pbimonitor
Linux Equivalent: /home/data/pbimonitor
Windows Path: C:\home\site\wwwroot\Scripts
Linux Equivalent: /home/site/wwwroot/Scripts

But I am not sure if I will encounter any problems down the line so I am deleting my resources and creating another one in windows.
I will edit the README file then create a PR.

Catalog - Workspaces RAW error evaluating group by

Large implementation (over 100k workspaces full scan, several days of 400+ modifications) using Azure blob storage is now failing on the group by:

Table.Group(#"Expanded Data.workspaces1", {"id"}, {{"AllRows", each _, type table [id=text, name=text, type=text, state=text, isOnDedicatedCapacity=logical, reports=list, dashboards=list, datasets=list, dataflows=list, ScanDate=nullable datetime]}}

)

Key Vault authentication & Azure deployment

Hi,

I tried to deploy the project on our Azure subscription from de .zip file, but the environnement variables are not created.
Does I have to include the config.json in the compress archive ?

Also, I didn't saw where the Key Vault is used in your scripts. Does I have to change anything to use this authentication method ?

Thanks for your sharing by the way, gold sources for Power BI administration. :)

Regards

pbilog

Hi,

Please could you advise where I can find the pbilog value.. is this something that needs creating?

Also do we use the appID and tenantID from the App Secret (App registration)

Upgrading to latest version: Any potential issues to watch out for?

Hi @RuiRomano,

We've been using pbimonitor at Des Moines University for a little over a year now, and it's been working absolutely flawlessly. Thank you so much for helping us set it up!

Earlier this year, we upgraded our original installation to the version published on 2022-02-01 (commit hash aadf0fc). That upgrade went off without a hitch, and everything is still working fine, but now I'm preparing to upgrade to the latest version again, and I've noticed that there have been a lot of changes since the last upgrade. Do you know if the latest pbimonitor reports will have any trouble combining batches of data that were generated by these different versions of the Azure Function scripts, or whether there are any other potential issues I should watch out for?

Thank you again, and I hope you and your family have a lovely holiday season! 😊

James

Limit Audits to a subscription

I am looking through the code to see if there is a way to limit the PBI Monitor to a specific subscription instead of pulling information from across the Tenant.

Load more detailed "Share" data to the report so we can see who shared what with who.

It would be good to change code for both PowerShell and Azure Functions ETL.

High memory usage when using local PowerShell solution

Hi @RuiRomano, I had implemented the local PowerShell solution for a large tenant in April. The solution was working fine till about August end, after which it was running into time-out issues. Trying to run the Power BI template on Power BI Desktop throws an error saying there is no available memory, even for machines as large as 678 GB RAM. Is there any way to optimize the data load to improve refresh performance, since incremental refresh is not possible on the local version of the solution?

Issue running in PowerShell

I have set this up just as described in the ReadME and keep running into this error: Fetch - Run.ps1 : Error on '.\Fetch - Activity.ps1' - System.AggregateException: One or more errors occurred. ---> System.Net.Http.HttpRequestException: Response status code does not indicate success: 401 (Unauthorized).

I have checked and double checked everything and cannot figure out the source of the error.

Tidy up the code

Comment about Datasetrefresh

Hi,

Thank you for for this amazing project, @RuiRomano!

It was very much ready out of the box, following your instructions per readme.md
use Azure Functions, and only had a few things I needed to customize:

I wanted a report page for Dataset Refreshes in the same report as the other stuff. That led to some tinkering like getting PowerQuery snippets, measures and calculated columns from the refreshes template (which was configured for disk as source) and merging it into the template for blobsource.
Dataset refreshes uses a non-admin API, which means the service principal needs to be admin/member of the workspaces where refreshes should be monitored. (this is natural, but was not commented in the readme).
PS: In my setup, service principal cannot access the standard API to get dataset refreshes by having a workspace role through a group. Has to have the role directly.

Power bi Performance

Hi, we have a problem when we need to load a tenant with lots of reports.
example :
workspaces: 10.000
reports: 6.000
users: 6.000
and we were running the solution ( with files) since november 2022 and we got to a point that power query stopped working for the catalog when combining files. Any idea how to improve the solution ?