Code Monkey home page Code Monkey logo

azureoptimizationengine's Introduction

Azure Optimization Engine

The Azure Optimization Engine (AOE) is an extensible solution designed to generate optimization recommendations for your Azure environment. See it like a fully customizable Azure Advisor. Actually, the first custom recommendations use-case covered by this tool was augmenting Azure Advisor Cost recommendations, particularly Virtual Machine right-sizing, with a fit score based on VM metrics and properties. Other recommendations are being added to the tool, not only for cost optimization but also for security, high availability and other Well-Architected Framework pillars. You are welcome to contribute with new types of recommendations!

It is highly recommended that you read the whole blog series dedicated to this project, starting here. You'll find all the information needed to understand the whole solution. Luke Murray wrote a great Azure Spring Clean 2022 blog post describing AOE and how to set it up, with full deployment and configuration steps.

README index

What you can get

A few hours after setting up the engine, you'll get a Power BI dashboard with all Azure optimization opportunities, coming from both Azure Advisor and from custom recommendations included in the engine. These recommendations are also available, with less detail, directly from the Azure Portal, via the Recommendations Workbook. Recommendations are updated every 7 days and you can contribute with your own custom ones if needed. Check below some examples of the Power BI dashboard and Recommendations Workbook pages.

Built-in custom recommendations

Besides collecting all Azure Advisor recommendations, AOE includes other custom recommendations that you can tailor to your needs, such as:

  • Cost
    • Augmented Advisor Cost VM right-size recommendations, with fit score based on Virtual Machine guest OS metrics (collected by Log Analytics agents) and Azure properties
    • Underutilized VM Scale Sets
    • Unattached disks
    • Standard Load Balancers without backend pool
    • Application Gateways without backend pool
    • VMs deallocated since a long time ago (forgotten VMs)
    • Orphaned Public IPs
    • Underutilized Azure SQL Databases (DTU-based SKUs only)
    • Storage Accounts without retention policy in place
    • Underutilized App Service Plans
    • App Service Plans without any application
    • Premium SSD disk has been underutilized
    • Virtual Machine is stopped (not deallocated) and still incurring costs
  • High Availability
    • Virtual Machine high availability (availability zones count, availability set, managed disks, storage account distribution when using unmanaged disks)
    • VM Scale Set high availability (availability zones count, managed disks)
    • Availability Sets structure (fault/update domains count)
  • Performance
    • VM Scale Sets constrained by lack of compute resources
    • SQL Databases constrained by lack of resources (DTU-based SKUs only)
    • App Service Plans constrained by lack of compute resources
  • Security
    • Service Principal credentials/certificates without expiration date
    • NSG rules referring to empty or inexisting subnets
    • NSG rules referring to orphan or removed NICs
    • NSG rules referring to orphan or removed Public IPs
  • Operational Excellence
    • Load Balancers without backend pool
    • Service Principal credentials/certificates expired or about to expire
    • Subscriptions close to the maximum limit of RBAC assignments
    • Management Groups close to the maximum limit of RBAC assignments
    • Subscriptions close to the maximum limit of resource groups
    • Subnets with low free IP space
    • Subnets with too much IP space wasted
    • Empty subnets
    • Orphaned NICs

Recommendations overview (Azure Workbook)

An overview of all your optimization recommendations

Cost opportunities overview (Azure Workbook)

An overview of your Cost optimization opportunities

Recommendations overview (Power BI)

An overview of all your optimization recommendations

Cost opportunities overview (Power BI)

An overview of your Cost optimization opportunities

Augmented VM right-size overview (Power BI)

An overview of your VM right-size recommendations

Fit score history for a specific recommendation (Power BI)

Fit score history for a specific recommendation

Log Analytics Workbooks

With AOE's Log Analytics Workbooks, you can explore many perspectives over the data that is collected every day. For example, costs growing anomalies, Microsoft Entra ID and Azure RM principals and roles assigned, how your resources are distributed, how your Block Blob Storage usage is distributed, how your Azure Benefits usage is distributed (supports only Enterprise Agreement customers) or exploring Azure Policy compliance results over time.

Costs growing anomalies

Virtual Machines perspectives over time

Microsoft Entra ID/Azure Resource Manager principals and roles summary, with service principal credentials expiration

Privileged Microsoft Entra ID roles and assignment history

Block Blob Storage usage analysis with Lifecycle Management recommendations

Azure Benefits usage analysis with a comparison between Reservations and On-Demand/Savings Plan prices

Policy Compliance state, with evolution over time

Releases

  • 12/2023
    • Added support for User-Assigned Managed Identity
    • Added tool for setting DCR-based (Azure Monitor Agent) performance counters collection
    • Changed deployment template from ARM to Bicep
    • Recommendations workbook: filtering by tags
    • New Cost recommendation added
      • Virtual Machine is stopped (not deallocated) and still incurring costs
    • New workbooks
      • Savings Plans Usage (preview)
    • Azure China consumption exports (preview)
    • Several bug fixes and performance, security, and usability improvements
  • 09/2023
    • Added support for MCA, CSP Modern and other customer agreement types
    • New Cost recommendation added
      • Premium SSD disk has been underutilized
    • New workbooks (see how to enable them)
      • Benefits Simulation
      • Benefits Usage
      • Reservations Potential
      • Reservations Usage
    • Support for silent deployment/upgrade and optional Workbooks deployment (thanks to @wpouseele)
    • Support for AOE resources tags assignment at deployment/upgrade time
    • Policy Compliance workbook: filtering/grouping by tags
    • Several bug fixes and performance and security improvements
  • 12/2022
    • Migrated from the deprecated Consumption API to the new Cost Details API
    • Recommendations report available also in Azure Workbook (besides Power BI)
    • New Cost recommendations added
      • Underutilized App Service Plans
      • App Service Plans without any application
    • New Performance recommendations added
      • SQL Databases constrained by lack of resources (DTU-based SKUs only)
      • App Service Plans constrained by lack of compute resources
    • New workbooks
      • Recommendations
      • Block Blob Storage Usage
      • Benefits Usage (preview, Enterprise Agreement customers only)
    • Several bug fixes and performance improvements
  • 06/2022
    • New Cost recommendations added
      • Underutilized SQL Databases (DTU-based SKUs only)
      • Storage Accounts without retention policy in place
    • New workbooks
      • Policy Compliance
      • Reservations Potential (preview)
      • Reservations Usage (preview)
    • Added Storage Lifecycle Management policy, cleaning up CSVs older than 6 months (new deployments or full upgrades only)
    • Added SQL Database retention policy runbook to clean up recommendations older than 365 days (configurable)
    • Support for (any type of) existing SQL Server reuse (upgrade scenarios only, after manually migrating original database to new server)
    • Workbooks usability improvements
    • Runbook performance improvements
    • Several bug fixes
  • 12/2021
    • Several new recommendations added
      • Cost - Underutilized VM Scale Sets
      • Performance - VM Scale Sets constrained by lack of compute resources
      • High Availability - VM Scale Sets not using managed disks
      • High Availability - Virtual Machines or VM Scale Sets not leveraging multiple Availability Zones
    • Microsoft Entra ID exports migrated to the Microsoft Graph SDK (preparing for the deprecation of Azure AD Graph)
    • Automation Runbooks use by default Managed Identity authentication (no more certificate-based Run As Accounts)
    • Support for resetting Automation schedules with the Reset-AutomationSchedules.ps1 script
    • Workbooks usability and content improvements
    • Runbook performance improvements
    • Several bug fixes
  • 09/2021
    • Several new recommendations added
      • Operational Excellence - Subscriptions close to the maximum limit of RBAC assignments
      • Operational Excellence - Management Groups close to the maximum limit of RBAC assignments
      • Operational Excellence - Subscriptions close to the maximum limit of resource groups
      • Operational Excellence - Subnets with low free IP space
      • Operational Excellence - Subnets with too much IP space wasted
      • Operational Excellence - Empty subnets
      • Operational Excellence - Orphan NICs
      • Cost - Orphan Public IPs
      • Security - NSG rules referring to empty or inexisting subnets
      • Security - NSG rules referring to orphan or removed NICs
      • Security - NSG rules referring to orphan or removed Public IPs
    • Two new remediations added
      • Downgrade long-deallocated VMs disks
      • Delete/downgrade unattached disks
    • Added Networking assets and RBAC assignments to the Resources Inventory Workbook
    • Simplified ARM template deployment
    • Upgraded PowerShell module dependencies
    • Merged VMs High Availability recommendations into a single runbook
    • Several bug fixes
  • 06/2021
    • Introduced Log Analytics Workbooks with additional insights besides Power BI recommendations:
      • Identities and Roles - Microsoft Entra ID principals (user and service) and roles assigned both in Microsoft Entra ID and Azure RM, credentials expiration (for service principals)
      • Resouces Inventory (overall, VMs, disks, all with an historical perspective)
      • Costs Growing - cost anomalies observed during a specific period
    • Support for partial AOE upgrades - useful in scenarios where the solution was customized (e.g., Private Link or Hybrid Worker setups or changes in Automation Variables)
    • Added daily Microsoft Entra ID and Azure RM RBAC assignments exports
    • Added daily resource containers exports (subscriptions and resource groups)
    • Added OS and image models to VM exports
    • Improved support for US Government cloud
    • Support for Azure subscription names in recommendations
    • SQL Database component changed from Serveless to cheaper Basic plan
    • Several bug fixes
  • 03/2021 - support for suppressions, new recommendations added and deployment improvements
    • Support for recommendations suppressions (exclude, dismiss, snooze)
    • Five new recommendations added
      • Cost - Standard Load Balancers without backend pool
      • Cost - Application Gateways without backend pool
      • Security - Service Principal credentials/certificates without expiration date
      • Operational Excellence - Service Principal credentials/certificates expired or about to expire
      • Operational Excellence - Load Balancers without backend pool
    • Helper script that checks whether Log Analytics workspaces are configured with the right performance counters (with auto-fix support)
    • Support for multiple Log Analytics workspaces (VM metrics ingestion by Log Analytics agents)
    • Last deployment options are stored locally to make upgrades/re-deployments easier
    • Roles assigned to the Automation Run As Account have now the least privileges needed
    • Recommendations Dismissed/Postponed in Azure Advisor are now filtered
    • Power BI report improvements
    • Several bug fixes
  • 01/2021 - solution deployment improvements and several new recommendations added
    • Support for Azure Cloud Shell (PowerShell) deployment
    • Solution upgrade keeps original runbook schedules
    • Eight new recommendations added
      • Cost - VMs that have been deallocated for a long time
      • High Availability - Availability Sets with a small fault domain count
      • High Availability - Availability Sets with a small update domain count
      • High Availability - Unmanaged Availability Sets with VMs sharing storage accounts
      • High Availability - Storage Accounts containing unmanaged disks from multiple VMs
      • High Availability - VMs without Availability Set
      • High Availability - Single VM Availability Sets
      • High Availability - VMs with unmanaged disks spanning multiple storage accounts
  • 12/2020 - added Azure Consumption dimension to cost recommendations and refactored Power BI dashboard
  • 11/2020 - support for automated VM right-size remediations and for other Well-Architected scopes, with unmanaged disks custom recommendation
  • 07/2020 - [initial release] Advisor Cost augmented VM right-size recommendations and orphaned disks custom recommendation

Architecture

The AOE runs mostly on top of Azure Automation and Log Analytics. The diagram below depicts the architectural components. For a more detailed description, please read the whole blog series dedicated to this project, starting here.

Azure Optimization Engine architecture

Deployment instructions

Requirements

  • A supported Azure subscription (see the FAQs)
  • Azure Powershell 6.6.0+
  • Microsoft.Graph.Authentication and Microsoft.Graph.Identity.DirectoryManagement PowerShell modules (version 2.4.0+)
  • A user account with Owner permissions over the chosen subscription, so that the Automation Managed Identity is granted the required privileges over the subscription (Reader) and deployment resource group (Contributor)
  • (Optional) A user account with at least Privileged Role Administrator permissions over the Microsoft Entra tenant, so that the Managed Identity is granted the required privileges over Microsoft Entra ID (Global Reader)
  • (Optional) A user account with administrative privileges over the Enterprise Agreement (Enterprise Enrollment Administrator) or the Microsoft Customer Agreement (Billing Profile Owner), so that the Managed Identity is granted the required privileges over your consumption agreement

During deployment, you'll be asked several questions. You must plan for the following:

  • Whether you're going to reuse an existing Log Analytics Workspace or a create a new one. IMPORTANT: you should ideally reuse a workspace where you have VMs onboarded and already sending performance metrics (Perf table), otherwise you will not fully leverage the augmented right-size recommendations capability. If this is not possible/desired for some reason, you can still manage to use multiple workspaces (see Configuring Log Analytics workspaces).
  • An Azure subscription to deploy the solution (if you're reusing a Log Analytics workspace, you must deploy into the same subscription the workspace is in).
  • A unique name prefix for the Azure resources being created (if you have specific naming requirements, you can also choose resource names during deployment)
  • Azure region
  • (Optional) Enterprise Agreement Billing Account ID (EA/MCA customers) and the Billing Profile IDs (MCA customers)

Bear in mind that the AOE deployment creates the following (you cannot reuse any existing resource, except the Resource Group and the Log Analytics workspace):

  • Azure Automation Account
  • Storage Account
  • Azure SQL Server + SQL Database
  • (optional) Log Analytics workspace
  • (optional) Resource Group

Installation

The simplest, quickest and recommended method for installing AOE is by using the Azure Cloud Shell (PowerShell). You just have to follow these steps:

  1. Open Azure Cloud Shell (PowerShell)
  2. Run git clone https://github.com/helderpinto/AzureOptimizationEngine.git azureoptimizationengine
  3. Run cd azureoptimizationengine
  4. (optional) Run Install-Module Microsoft.Graph.Authentication,Microsoft.Graph.Identity.DirectoryManagement - this is required to grant the Global Reader role to the Automation Managed Identity in Microsoft Entra ID
  5. Run .\Deploy-AzureOptimizationEngine.ps1
  6. Input your deployment options and let the deployment finish (it will take less than 5 minutes)

If the deployment fails for some reason, you can simply repeat it, as it is idempotent. The same if you want to upgrade a previous deployment with the latest version of the repo. You just have to keep the same deployment options. Cool feature: the deployment script persists your previous deployment options and lets you reuse it!

If you don't want to use Azure Cloud Shell and prefer instead to run the deployment from your workstation's file system, you must first install the Az Powershell module (instructions here) and also the Microsoft.Graph modules (instructions here).

Optionally, you can specify the set of tags you want to assign to your AOE resources, by using the ResourceTags input parameter. For example:

$tags = @{"Service"="aoe";"Environment"="Demo"}
.\Deploy-AzureOptimizationEngine.ps1 -ResourceTags $tags

If you choose to deploy all the dependencies from your own local repository, you must publish the solution files into a publicly reachable URL. You must ensure the entire AOE project structure is available at the same base URL. Storage Account SAS Token-based URLs are not supported.

.\Deploy-AzureOptimizationEngine.ps1 -TemplateUri <URL to the Bicep file (e.g., https://contoso.com/azuredeploy.bicep)> [-AzureEnvironment <AzureUSGovernment|AzureGermanCloud|AzureCloud>]

# Example - Deploying from a public endpoint
.\Deploy-AzureOptimizationEngine.ps1 -TemplateUri "https://contoso.com/azuredeploy.bicep"

# Example 2 - Deploying from a public endpoint, using resource tags
$tags = @{"CostCenter"="FinOps";"Environment"="Production"}
.\Deploy-AzureOptimizationEngine.ps1 -TemplateUri "https://contoso.com/azuredeploy.bicep" -ResourceTags $tags

Optionally, you can also use the SilentDeploymentSettingsPath input parameter to deploy AOE in a more automated way.
The file referencing should be a JSON file with the needed attributes defined.
An example of the content of such silent deployment file is:

{
    "SubscriptionId": "<<SubscriptionId>>",                         // mandatory, subscription where AOE needs to be deployed
    "NamePrefix": "<<CustomNamePrefix>>",                           // mandatory, prefix for all resources. Fill in 'EmptyNamePrefix' to specify the resource names
    "WorkspaceReuse": "n",                                          // mandatory, y/n, y = reuse existing workspace, n = create new workspace
    "ResourceGroupName": "<<CustomName>>-rg",                       // mandatory if NamePrefix is set to 'EmptyNamePrefix', otherwise optional
    "StorageAccountName": "<<CustomName>>sa",                       // mandatory if NamePrefix is set to 'EmptyNamePrefix', otherwise optional
    "AutomationAccountName": "<<CustomName>>-auto",                 // mandatory if NamePrefix is set to 'EmptyNamePrefix', otherwise optional
    "SqlServerName": "<<CustomName>>-sql",                          // mandatory if NamePrefix is set to 'EmptyNamePrefix', otherwise optional
    "SqlDatabaseName": "<<CustomName>>-db",                         // mandatory if NamePrefix is set to 'EmptyNamePrefix', otherwise optional
    "WorkspaceName": "<<ExistingName>>",                            // mandatory if workspaceReuse is set to 'n', otherwise optional
    "WorkspaceResourceGroupName": "<<ExistingName>>",               // mandatory if workspaceReuse is set to 'n', otherwise optional
    "DeployWorkbooks": "y",                                         // mandatory, y/n, y = deploy the workbooks, n = don't deploy the workbooks
    "SqlAdmin": "<<sqlaAdmin>>",                                    // mandatory
    "SqlPass": "<<sqlPass>>",                                       // mandatory
    "TargetLocation": "westeurope",                                 // mandatory
    "DeployBenefitsUsageDependencies": "y",                         // mandatory, y/n, deploy the dependencies for the Azure Benefits usage workbooks (EA/MCA customers only + agreement administrator role required
    "CustomerType": "MCA",                                          // mandatory if DeployBenefitsUsageDependencies is set to 'y', MCA/EA
    "BillingAccountId": "<guid>:<guid>_YYYY-MM-DD",                 // mandatory if DeployBenefitsUsageDependencies is set to 'y', MCA or EA Billing Account ID
    "BillingProfileId": "ABCD-DEF-GHI-JKL",                         // mandatory if CustomerType is set to 'MCA", otherwise optional
    "CurrencyCode": "EUR"                                           // mandatory if DeployBenefitsUsageDependencies is set to 'y', EUR/USD/...
  }
  

Upgrading AOE

If you have a previous version of AOE and wish to upgrade, it's as simple as re-running the deployment script with the resource naming options you chose at the initial deployment. It will re-deploy the ARM template, adding new resources and updating existing ones.

However, if you previously customized components such as Automation variables or schedules, improved job execution performance with Hybrid Workers, or hardened the solution with Private Link, then you should run the deployment script with the DoPartialUpgrade switch, e.g.:

.\Deploy-AzureOptimizationEngine.ps1 -DoPartialUpgrade

With the DoPartialUpgrade switch, the deployment will only:

  • Add new storage containers
  • Update/add Automation runbooks
  • Update/add Automation modules
  • Add new Automation schedules
  • Add new Automation variables
  • Upgrade the SQL database model
  • Update Log Analytics Workbooks

Some customers may also customize the SQL Server deployment, for example, migrating from SQL Database to a SQL Managed Instance. There is no tooling available to assist in the migration, but once the database migration is done manually, the AOE upgrade script supports future DoPartialUpgrade upgrades with the IgnoreNamingAvailabilityErrors switch on (skips SQL Server naming/existence validation).

Usage instructions

Once successfully deployed, and assuming you have your VMs onboarded to Log Analytics and collecting all the required performance counters, we have everything that is needed to start augmenting Advisor recommendations and even generate custom ones! The first recommendations will be available more or less 3h30m after the deployment. In order to see them, you just have to open the Recommendations Workbook. For a more detailed analysis, you'll need to connect Power BI to the AOE database (see details below). Every week at the same time, AOE recommendations will be updated according to the current state of your environment.

Validating whether Log Analytics is collecting the right performance counters

To ensure the VM right-size recommendations have all the required data to provide its full value, Log Analytics must be collecting specific performance counters. Depending on the type of agent your are using to monitor your machines (Log Analytics agent or Azure Monitor Agent) you can use a helper tool for a quick configuration of the pre-requisites. For more details, see Configuring Log Analytics workspaces.

Widening the scope of AOE recommendations - more subscriptions or more workspaces

By default, the Azure Automation Managed Identity is assigned the Reader role only over the respective subscription. However, you can widen the scope of its recommendations just by granting the same Reader role to other subscriptions or, even simpler, to a top-level Management Group.

In the context of augmented VM right-size recommendations, you may have your VMs reporting to multiple workspaces. If you need to include other workspaces - besides the main one AOE is using - in the recommendations scope, you just have to add their workspace IDs to the AzureOptimization_RightSizeAdditionalPerfWorkspaces variable (see more details in Configuring Log Analytics workspaces).

Adjusting AOE thresholds to your needs

For Advisor Cost recommendations, the AOE's default configuration produces percentile 99th VM metrics aggregations, but you can adjust those to be less conservative. There are also adjustable metrics thresholds that are used to compute the fit score. The default thresholds values are 30% for CPU (5% for shutdown recommendations), 50% for memory (100% for shutdown) and 750 Mbps for network bandwidth (10 Mbps for shutdown). All the adjustable configurations are available as Azure Automation variables (for example, AzureOptimization_PerfPercentile* and AzureOptimization_PerfThreshold*).

For all the available customization details, check Customizing the Azure Optimization Engine.

Changing the schedules for the exports and recommendations runbooks

By default, the base time for the AOE Automation schedules is set as the deployment time. Soon after the initial deployment completes, the exports, ingests and recommendations runbooks will run according to the engine's default schedules. For example, if you deploy AOE on a Monday at 11:00 a.m., you will get new recommendations every Monday at 2:30 p.m.. If this schedule, for some reason, does not fit your needs, you can reset it to the time that better suits you, by using the Reset-AutomationSchedules.ps1 script. You just have to call the script following the syntax below and answer the input requests:

./Reset-AutomationSchedules.ps1 -AutomationAccountName <AOE automation account> -ResourceGroupName <AOE resource group> [-AzureEnvironment <AzureUSGovernment|AzureGermanCloud|AzureCloud>]

The base time you choose must be in UTC and must be defined according to the week day and hour you want recommendations to be generated. You must deduce 3h30m from the time you choose, because the base time defines the schedules for all the dependent automation runbooks that must run before the recommendations are generated. For example, let's say you want recommendations to be generated every Monday at 8h30 a.m.; the base time will be the next calendar date falling on a Monday, at 5h00 a.m.. The format of the date you choose must be YYYY-MM-dd HH:mm:ss, e.g., 2022-01-03 05:00:00.

The script will also ask you to enter, if needed, the Hybrid Worker Group you want the runbooks to run in (see the next sub-section).

Changing the runbooks execution context to Hybrid Worker

By default, AOE Automation runbooks are executed in the context of the Azure Automation sandbox. If you face performance issues due to the memory limits of the Automation sandbox or decide to implement private endpoints for the Storage Account or SQL Database, to harden AOE's security, you will need to execute runbooks from a Hybrid Worker (an Azure or on-premises Virtual Machine with the Automation Hybrid Worker extension). To change the execution context for the AOE runbooks, you must use the Reset-AutomationSchedules.ps1 script. See how to use the script in the previous sub-section and, after setting the runbooks execution base time, enter the Hybrid Worker Group name you want the runbooks to run in.

IMPORTANT:

  • The Hybrid Worker machine must have the required PowerShell modules installed. See here the list of required modules.
  • Once you change the runbook execution context to Hybrid Worker, you will have to always use the DoPartialUpgrade flag whenever you upgrade AOE, or else you will lose the runbook schedule settings and revert to the default sandbox configuration.
  • The Managed Identity used to authenticate against Azure, Microsoft Entra ID and Billing Account scopes is still the Azure Automation's one, even if the Hybrid Worker machine has a Managed Identity assigned (see details). User-assigned Managed Identities are supported in the context of Hybrid Workers only if 1) the Automation Account does not have any associated Managed Identity, i.e., only the Hybrid Worker machine can have a User-Assigned Managed Identity; 2) all runbooks run in the context of the Hybrid Worker. In this case, you must create an AzureOptimization_UAMIClientID Automation Variable with the User-Assigned Managed Identity Client ID as value; and 3) the AzureOptimization_AuthenticationOption Automation variable value is updated to UserAssignedManagedIdentity.

Visualizing recommendations with Power BI

The AOE includes a Power BI sample report for visualizing recommendations. To use it, you have first to change the data source connection to the SQL Database you deployed with the AOE. In the Power BI top menu, choose Transform Data > Data source settings.

Open the Transform Data > Data source settings menu item

Then click on "Change source" and change to your SQL database server URL (don't forget to ensure your SQL Firewall rules allow for the connection).

Click on Change source and update SQL Server URL

If the connection fails at the first try, this might be because the SQL Database was paused (it was deployed in the cheap Serverless plan). At the next try, the connection should open normally.

The report was built for a scenario where you have an "environment" tag applied to your resources. If you want to change this or add new tags, open the Transform Data menu again, but now choose the Transform data sub-option. A new window will open. If you click next in "Advanced editor" option, you can edit the data transformation logic and update the tag processing instructions.

Open the Transform Data > Transform data menu item, click on Advanced editor and edit accordingly

Suppressing recommendations

If some recommendation is not applicable or you want it to be removed from the report while you schedule its mitigation, you can suppress it, either for a specific resource, resource group, subscription or even solution-wide. See Suppressing recommendations for more details.

Enabling the Reservations and Benefits Usage workbooks

In order to leverage the Workbooks that allow you to analyze your Azure benefits usage (Benefits Usage and Reservations Usage) or estimate the impact of doing additional consumption commitments (Benefits Simulation and Reservations Potential), you need to configure AOE and grant privileges to its Managed Identity at your consumption agreement level (EA or MCA). If you could not do it during setup/upgrade, you can still execute those extra configuration steps, provided you do it with a user that is both Contributor in the AOE resource group and have administrative privileges over the consumption agreement (Enterprise Enrollment Administrator for EA or Billing Profile Owner for MCA). You just have to use the Setup-BenefitsUsageDependencies.ps1 script following the syntax below and answer the input requests:

./Setup-BenefitsUsageDependencies.ps1 -AutomationAccountName <AOE automation account> -ResourceGroupName <AOE resource group> [-AzureEnvironment <AzureUSGovernment|AzureGermanCloud|AzureCloud>]

If you run into issues with the Azure Pricesheet ingestion (due to the large size of the CVS export), you can create the following Azure Automation variable, to filter in the Price Sheet regions: AzureOptimization_PriceSheetMeterRegions set to the comma-separated billing regions of your virtual machines (e.g. EU West,EU North).

The Reservations Usage Workbook has a couple of "Unused Reservations" tiles that require AOE to export Consumption data at the EA/MCA scope (instead of the default Subscription scope). You can switch to EA/MCA scope consumption by creating/updating the AzureOptimization_ConsumptionScope Automation variable with BillingAccount as value. Be aware that this option may generate a very large single consumption export which may lead to errors due to lack of memory (this would in turn require deploying AOE with a Hybrid Worker).

Frequently Asked Questions

  • Is the AOE supported by Microsoft? No, the Azure Optimization Engine is not supported under any Microsoft standard support program or service. The scripts are provided AS IS without warranty of any kind. The entire risk arising out of the use or performance of the scripts and documentation remains with you.

  • What type of Azure subscriptions/clouds are supported? AOE has been deployed and tested against EA, MCA and MSDN subscriptions in the Azure commercial cloud (AzureCloud). Although not tested yet, it should also work in MOSA subscriptions. It was designed to also operate in the US Government cloud, though it was never tested there. Sponsorship (MS-AZR-0036P and MS-AZR-0143P), CSP (MS-AZR-0145P, MS-AZR-0146P, and MS-AZR-159P) DreamSpark (MS-AZR-0144P) and Internal subscriptions should also work, but due to lack of availability or disparities in their consumption (billing) exports models, some of the Workbooks may not fully work.

  • Why is my Power BI report empty? Most of the Power BI report pages are configured to filter out recommendations older than 7 days. If it shows empty, just try to refresh the report data.

  • Why is my VM right-size recommendations overview page empty? The AOE depends on Azure Advisor Cost recommendations for VM right-sizing. If no VMs are showing up, try increasing the CPU threshold in the Azure Advisor configuration... or maybe your infrastructure is not oversized after all!

  • Why are my VM right-size recommendations showing up with so many Unknowns for the metrics thresholds? The AOE depends on your VMs being monitored by Log Analytics agents and configured to send a set of performance metrics that are then used to augment Advisor recommendations. See more details here.

  • Why am I getting values so small for costs and savings after setting up AOE? The Azure consumption exports runbook has just begun its daily execution and only got one day of consumption data. After one month - or after manually kicking off the runbook for past dates -, you should see the correct consumption data.

  • What is the currency used for costs and savings? The currency used is the one that is reported by default by the Azure Consumption APIs. It should match the one you usually see in Azure Cost Management.

  • What is the default time span for collecting Azure consumption data? By default, the Azure consumption exports daily runbook collects 1-day data from 7 days ago. This offset works well for many types of subscriptions. If you're running AOE in PAYG or EA subscriptions, you can decrease the offset by adjusting the AzureOptimization_ConsumptionOffsetDays variable. However, using a value less than 2 days is not recommended.

  • Why is AOE recommending to delete a long-deallocated VM that was deallocated just a few days before? The LongDeallocatedVms recommendation depends on accurate Azure consumption exports. If you just deployed AOE, it hasn't collected consumption long enough to provide accurate recommendations. Let AOE run at least for 30 days to get accurate recommendations.

  • Why is AOE recommending to delete a long-deallocated VM that was already deleted? Due to the fact that Azure consumption exports are collected with a (default) offset of 7 days, the LongDeallocatedVms recommendation might recommend for deletion a long-deallocated VM that was meanwhile deleted. That false positive should normally disappear in the next iteration.

  • How much does running AOE in my subscription cost? The default AOE deployment requires cheap Azure resources: an Azure Automation account, a small Azure SQL Database and a Storage Account. It also requires a Log Analytics Workspace to which it sends all the logs used by Workbooks. The costs, mostly coming from Azure Automation, depend on the size of your environment, but even in customers with hundreds of VMs, AOE costs do not amount to more than 100 EUR/month. In small to medium-sized environments, it will cost less than 20 EUR/month. These costs do not include agent-based VM performance metrics ingestion into Log Analytics.

azureoptimizationengine's People

Contributors

cloudchristoph avatar helderpinto avatar wpouseele avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

azureoptimizationengine's Issues

Run-Connect-azaccount to login issue

working in azure us government cloud
Attempting to install via azure cloud shell (powershell)
Running Original script gets...

WARNING: Interactive authentication is not supported in this session, please run cmdlet 'Connect-AzAccount -UseDeviceAuthentication'.
Getting Azure subscriptions...
Get-AzSubscription: /home/blah/azureoptimizationengine/DeployAOE.ps1:142
Line |
142 | $subscriptions = Get-AzSubscription | Where-Object { $_.State -eq "En …
| ~~~~~~~~~~~~~~~~~~
| Run Connect-AzAccount to login.

If I make a few Tweaks and add the requested -usedeviceauthentication, it eventually gets to this...

New-AzADServicePrincipal: /home/blah/azureoptimizationengine/Deploy-AzureOptimizationEngine.ps1:54
Line |
54 | … Principal = New-AzADServicePrincipal -ApplicationId $Application.Appl …
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| The appId 'd5e41e0f-6273-4b73-a335-fc7e960712b6' of the service principal does not reference a valid application object.

The query on the Azure AD Roles does not return updated data

We have tried to find out why it does not bring the updated information in the workbook: "(Identities and Roles)" of our tenant when we enable it for the first time, however, when detecting that there is no overflow in the database and that the Jobs in the Automation Account -> runbooks are executing correctly and we do not know why it is not updated.

MicrosoftTeams-image
MicrosoftTeams-image (1)

Fully customizable naming for all Azure resources

Besides the current possibility of just choosing a naming prefix at deployment time, I also want to be able to choose the naming for all the Azure resources that are deployed as part of the solution (Log Analytics Workspace, Automation Account, Azure SQL Server, Azure SQL Database and Storage Account). This is required by enterprise scenarios with an existing naming convention.

Feature request

I really like the concept of this, but I would like to deploy this at resource group level and not at subscription scope.. I have been looking around but can't seem to change it, is there anyone from team who can assist?

Deployment failing for microsoft.operationalinsights/workspace

"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.","details":[{"code":"ResourceDeploymentFailure","message":"Encountered internal server error. Diagnostic information: timestamp '20220111T025224Z', subscription id 'a5197234-610b-4f87-8f44-a8665e7e8a9d', tracking id '16690699-2899-45e3-9bb0-2b5e2e9bbcd6', request correlation id '0c2d7113-c7a3-4d05-b2aa-e6565501cf3c'."},{"code":"UnsupportedApiVersion","message":"The HTTP resource that matches the request URI 'http://amsweb.amsweb/subscriptions/a5197234-610b-4f87-8f44-a8665e7e8a9d/resourcegroups/arg-syd-platform-mgmt-management/providers/microsoft.operationalinsights/workspaces/law-syd-platform-mgmt-01' does not support the API version '2021-12-01-preview'."}]}

Job failure with date_s exception

Hello ,

Thankx a lot for the new workbooks however few of the jobs have been failing on query to run .
Manual run of the query shows they fail to map "date_s" and LA worksspace dosent have table with a name "date_s"
would be able to help or point in right direction .

Thankx
Bagel

Work Analytics Workspace workbooks?

looking over the workbooks in the log analytics workspace, two work, but a third doesn't appear to ..
Is this something I need to set and haven't seen in the documentation because I am silly?

This query could not run because some parameters are not set.
Please set: SelectedPerspective, SelectedPerspectiveId

image

Get-AzConsumptionPriceSheet

Hello again,

Utilizing azure us government, with Get-AzConsumptionPriceSheet in Recommend-AdvisorCostAugmentedToBlobStorage

noticed that get-azconsumptionpricesheet returns:
Get-AzConsumptionPriceSheet: Operation returned an invalid status code 'NotFound'

Id :
Name :
PriceSheets : {}
Tag :
Type :

hmnm, ran it with -debug
got a snippit

"error": {
"code": "404",
"message": "Cost management data is not supported for subscription(s) xxx in the provided api-version. Please use api-version 2019-10-01 or later. (Request ID: yyy)"

going to do more research on rights to subscription cost management and azure us government

Update: MicrosoftDocs/azure-docs#38660
Changing the API version to the previous stable release (2019-01-01) allows the filter to work.
This is an example of the full URI: https://management.azure.com/subscriptions/00000000-0000-0000-0000-000000000000/providers/Microsoft.Consumption/usageDetails?api-version=2019-05-01&$filter=properties/resourceGroup eq 'dev-rg'

where the -debug of the command reported using
Absolute Uri:
https://management.usgovcloudapi.net/subscriptions/xxx/providers/Microsoft.Consumption
/pricesheets/default?$top=1000&api-version=2018-01-31

Power BI File Date Filter Issue

On the 7th Day of a recommendation cycle, prior to the weekly Recommendation runs, the Power BI View filters out the Week Old Data, and does not have new data to replace it with...

This Test Installation within a subscription has a first run on 6/22, second run on 6/29, and todays run has yet to occur
I have verified all the current recommendation data is in the power BI file
image
image

Note: I know that I can change the schedule to go after hours or before I start using the views etc, but this introduces a window of time your view will not work if you setup the wrong scheduling.

Automation Account runbook "Export-PolicyComplianceToBlobStorage" - Exception calling "Add" with "2" argument(s): "Item has already been added. Key in dictionary

The runbook Export-PolicyComplianceToBlobStorage of the Automation Account is failing with an error Exception calling "Add" with "2" argument(s): "Item has already been added. Key in dictionary.
This seems to be related to the Get-AzPolicyDefinition.
I checked it locally on my computer and I saw that some $definitions not having the DisplayName-property.
Regarding to this, the if-clause if (-not($policyInitiatives[$initiative.PolicySetDefinitionId])) with the not-parameter will be $true as it has no value and the script try to add the key/value which already exists.

I was using the code with the following command successfully:

    $definitions = Get-AzPolicyDefinition
    foreach ($definition in $definitions)
    {
        if (-not($policyDefinitions[$definition.PolicyDefinitionId]))
        {
            try {
                $displayName = $definition.Properties.DisplayName
                if([string]::IsNullOrWhiteSpace($displayName)) {
                    $policyDefinitions.Add($definition.PolicyDefinitionId, 'NotGiven')
                }
                else  {
                    $policyDefinitions.Add($definition.PolicyDefinitionId, $displayName)
                }
            }
            catch {
                Write-Warning "PolicyDefinitionId: $($definition.PolicyDefinitionId), DisplayName: $($definition.Properties.DisplayName)"
            }
        }
    }

Ability to dismiss or snooze recommendations

For Advisor recommendations, there should be a UI solution for invoking the Advisor API to dismiss/snooze a particular recommendation.

For AOE custom recommendations, we should use the same UI solution, but with our own data model and logic.

Should be implemented in conjunction with #13

Parameter Minimum TLS version of SQL server is defined

Hello,

After the deployment the parameter minimum TLS version of SQL server is not set/defined.
It generates TLS alerts if you have policies or CSPM in place.

AOE-TLS-Not-Set

Are you able to set/configure it to TL1.2 during the deployment?

Thank you!

Replace Az Advisor right-size target SKUs with customer-defined SKU families

Let's say Az Advisor recommends downsizing to a SKU a customer does not want to use for whatever reason (Reservations policy, performance requirements, etc.). AOE should allow customers to provide a list of allowed SKU families, from which Az Advisor recommendations would be updated to use a SKU from those families instead.

Request to add VM Resizing Recommendation instead of Augmenting Advisor

Azure Advisor not giving VM resizing Recommendations as per the docs, we have created few VMs with size Standard D4as v5 (4 vcpus, 16 GiB memory), which was running without any application ( max CPU utilisation < 6%). Still after 3 weeks no advisor recommendation.

As per the doc https://learn.microsoft.com/en-in/azure/advisor/advisor-cost-recommendations#resize-sku-recommendations , we should get the recommendation.

Can you kindly include VM resizing recommendation as well, instead of augmenting advisor.

No Right-Size data being populated

Hi All,

Hoping somebody can assist me with this. I am having trouble with no information coming through for the VM right sizing (page shows blank) and in general there are less recommendations than expected. Additionally, the Azure Advisor right-sizing recommendations are not present here either (not sure if that is by design or not). I changed the date range to be 1 month and also refreshed many times over a period of 2 weeks

From what I can see, all the subscriptions appear to be visible to the AOE, because I can see some recommendations for each subscription (i.e. Subnets without any IP usage are a waste of virtual network address space)

During the setup, I noticed our customer already had some VMs attached to log analytics workspaces, I can confirm that I added the AzureOptimization_RightSizeAdditionalPerfWorkspaces variable to the automation account and added all of the workspaces to this 2 weeks ago.

Also, I ensured that the Automation App account has reader access to the tenant root group which has reader access to all subscriptions and log workspaces.

Additional Info;

  • I am using the provided PowerBI sample report
  • I Have run the "./Setup-LogAnalyticsWorkspaces.ps1 -AutoFix" for all workspaces
  • There are 10 subscriptions in total, all of which are PAYG
  • The AOE has been running for 3 weeks as of time of writing (15-09-2022)

not having recommendations from all subscriptions

I have implemented the AOE for one Azure tenant which has multiple subscriptions, and all the VMs are reporting to a single LogAnalytics Workspace. I already have set up permission for the Managed Identity at Management group scope, but when I'm trying to execute queries against the recommendations, I only receive recommendations from the subscriptions where I installed the AOE.

What I'm doing wrong?

Cost Growing Workbook not showing any data

Costs Growing workbook is not showing any data. Below error is getting. Any suggesstion

Evolution over time (select growing/outlier cost)
This query could not run because some parameters are not set.
Please set: SelectedPerspective, SelectedPerspectiveId

Top contributing instances over time
This query could not run because some parameters are not set.
Please set: SelectedPerspective, SelectedPerspectiveId

Top contributing resource groups over time
This query could not run because some parameters are not set.
Please set: SelectedPerspective, SelectedPerspectiveId
cost_growing1

Automate remediation for Advisor right-size recommendations

Periodic remediation for Advisor right-size recommendations:

  • Accepts: confidence score and recommendation history thresholds, list of tag name-value pairs
  • Considers: right-size recommendations greater or equal than a given confidence score, being generated at least for a given number of weeks in a row, (optionally) for VMs with specific tags
  • Acts: for every VM meeting the remediation condition, change size to target SKU

Certain Recommend runbooks failing to run.

Hi There,

Hoping somebody can assist with this, I have several runbooks failing, these are listed below;

Recommend-VMSSOptimizationsToBlobStorage
Recommend-VMsHighAvailabilityToBlobStorage
Recommend-UnusedLoadBalancersToBlobStorage
Recommend-UnusedAppGWsToBlobStorage
Recommend-SqlDbOptimizationsToBlobStorage

All of these have similar errors, please see two examples below;

EXAMPLE 1:

Recommend-VMsHighAvailabilityToBlobStorage

Output:

Logging in to Azure with ManagedIdentity...
Environments

{[AzureChinaCloud, AzureChinaCloud], [AzureCloud, AzureCloud], [AzureGermanCloud, AzureGermanCloud], [AzureUSGovernme...
Finding tables where recommendations will be generated from...
Will run query against tables AzureOptimizationAvailSetsV1_CL, AzureOptimizationVMsV1_CL, AzureOptimizationVMSSV1_CL, AzureOptimizationVhdDisksV1_CL and AzureOptimizationResourceContainersV1_CL
Name : Internal IT (xxxxx) - xxxxx -
MSI@xxxxx
Account : MSI@xxxxx
Environment : AzureCloud
Subscription : xxxxx
Tenant : xxxxx
TokenCache :
VersionProfile :
ExtendedProperties : {}
Looking for Availability Sets with a low fault domain count...

Warnings:

-TenantId ‘xxxxx’ contains more than one active subscription. First one will be selected for further use. To select another subscription, use Set-AzContext.

-Query failed. Debug the following query in the AOE Log Analytics workspace: AzureOptimizationAvailSetsV1_CL | where TimeGenerated > ago(1d) and toint(FaultDomains_s) < 3 and toint(FaultDomains_s) < todouble(VmCount_s)/2 | project TimeGenerated, InstanceId_s, InstanceName_s, ResourceGroupName_s, SubscriptionGuid_g, TenantGuid_g, Cloud_s, Tags_s, FaultDomains_s, VmCount_s | join kind=leftouter ( AzureOptimizationResourceContainersV1_CL | where TimeGenerated > ago(1d) | where ContainerType_s =~ 'microsoft.resources/subscriptions' | project SubscriptionGuid_g, SubscriptionName = ContainerName_s ) on SubscriptionGuid_g

-Operation returned an invalid status code 'BadRequest'

EXAMPLE 2:

Recommend-VMSSOptimizationsToBlobStorage

Output:

Logging in to Azure with ManagedIdentity...
Environments

{[AzureChinaCloud, AzureChinaCloud], [AzureCloud, AzureCloud], [AzureGermanCloud, AzureGermanCloud], [AzureUSGovernme...
Finding tables where recommendations will be generated from...
Will run query against tables AzureOptimizationVMSSV1_CL, AzureOptimizationMonitorMetricsV1_CL, AzureOptimizationResourceContainersV1_CL and AzureOptimizationConsumptionV1_CL
Name : Internal IT (xxxxx) - xxxxx -
MSI@xxxxx
Account : MSI@xxxxx
Environment : AzureCloud
Subscription : xxxxx
Tenant : xxxxx
TokenCache :
VersionProfile :
ExtendedProperties : {}
Getting Virtual Machine SKUs for the australiaeast region...
Getting the current Pricesheet...
Consumption pricesheet not available, will estimate savings based in cores count...
Looking for underutilized Scale Sets, with less than 30% CPU and 50% RAM usage...

Warnings:

-TenantId 'xxxxx' contains more than one active subscription. First one will be selected for further use. To select another subscription, use Set-AzContext.

-SubscriptionGuid_g, Cloud_s, TenantGuid_g, VMSSSize_s, NicCount_s, DataDiskCount_s, PMemoryPercentage, PCPUPercentage, Tags_s, Last30DaysCost, Last30DaysQuantity | join kind=leftouter ( AzureOptimizationResourceContainersV1_CL | where TimeGenerated > ago(1d) | where ContainerType_s =~ 'microsoft.resources/subscriptions' | project SubscriptionId = SubscriptionGuid_g, SubscriptionName = ContainerName_s ) on SubscriptionId | where isnotempty(PMemoryPercentage) and isnotempty(PCPUPercentage) and PMemoryPercentage < 50 and PCPUPercentage < 30

-Error getting consumption data: Cost Management not available (Cost management data is not supported for subscription(s) xxxxx in the provided api-version. Please use api-version 2019-10-01 or later. (Request ID: e3cb353c-c42b-4b71-afa1-234e815d4481)). 1 of 3 tries. Waiting 30 seconds...

-Query failed. Debug the following query in the AOE Log Analytics workspace: let billingInterval = 30d; let perfInterval = 7d; let cpuPercentileValue = 99; let memoryPercentileValue = 99; let etime = todatetime(toscalar(AzureOptimizationConsumptionV1_CL | summarize max(UsageDate_t))); let stime = etime-billingInterval; let BilledVMs = AzureOptimizationConsumptionV1_CL | where UsageDate_t between (stime..etime) and InstanceId_s contains 'virtualmachinescalesets' | extend VMConsumedQuantity = iif(InstanceId_s contains 'virtualmachinescalesets' and MeterCategory_s == 'Virtual Machines', todouble(Quantity_s), 0.0) | extend VMPrice = iif(InstanceId_s contains 'virtualmachinescalesets' and MeterCategory_s == 'Virtual Machines', todouble(UnitPrice_s), 0.0) | extend FinalCost = VMPrice * VMConsumedQuantity | summarize Last30DaysCost = sum(FinalCost), Last30DaysQuantity = sum(VMConsumedQuantity) by InstanceId_s; let MemoryPerf = AzureOptimizationMonitorMetricsV1_CL | where TimeGenerated > ago(perfInterval) | where MetricNames_s == "Available Memory Bytes" and AggregationType_s == "Minimum" | extend MemoryAvailableMBs = todouble(MetricValue_s)/1024/1024 | project TimeGenerated, MemoryAvailableMBs, InstanceId_s=ResourceId | join kind=inner ( AzureOptimizationVMSSV1_CL | where TimeGenerated > ago(1d) | distinct InstanceId_s, MemoryMB_s ) on InstanceId_s | extend MemoryPercentage = todouble(toint(MemoryMB_s) - toint(MemoryAvailableMBs)) / todouble(MemoryMB_s) * 100 | summarize PMemoryPercentage = percentile(MemoryPercentage, memoryPercentileValue) by InstanceId_s; let ProcessorPerf = AzureOptimizationMonitorMetricsV1_CL | where TimeGenerated > ago(perfInterval) | where MetricNames_s == "Percentage CPU" and AggregationType_s == 'Maximum' | extend InstanceId_s = ResourceId | summarize PCPUPercentage = percentile(todouble(MetricValue_s), cpuPercentileValue) by InstanceId_s; AzureOptimizationVMSSV1_CL | where TimeGenerated > ago(1d) | distinct InstanceId_s, VMSSName_s, ResourceGroupName_s, SubscriptionGuid_g, Cloud_s, TenantGuid_g, VMSSSize_s, NicCount_s, DataDiskCount_s, Tags_s | join kind=inner ( BilledVMs ) on InstanceId_s | join kind=leftouter ( MemoryPerf ) on InstanceId_s | join kind=leftouter ( ProcessorPerf ) on InstanceId_s | project InstanceId_s, VMSSName = VMSSName_s, ResourceGroup = ResourceGroupName_s, SubscriptionId =

-Operation returned an invalid status code 'BadRequest'

Doubt in logic of metrics exporting

Hi Luke,

Correct me if am wrong, for metrics collection using az monitor command we have option to look back for max 30/90 days to analyse utilisation.
Then why we are exporting daily utilisation to log analytics workspace and analysing average utilisation using kql and variable (look back period).

SQL ingestion failing

Hello the weekly SQL ingestion job starts failing after first run
Looks like the job is consuming all the 400 MB memory allocated to it .does a anyone has any work around ?

Job Name:Ingest-RecommendationsToSQLServer
Logs:The runbook job failed due to sandbox running out of memory. Each Azure Automation sandbox is allocated 400 MB of memory. The job was attempted 3 times before it was suspended. See some common ways to resolve this at https://aka.ms/AAMemoryLimit

Support for CSP subscriptions

Discussed in #69

Originally posted by spike28742 March 22, 2023
Hi

When I try to deploy I am getting an error that the subscription is not supported. I managed to deploy the solution about a month ago. Are there any limitations why the solution will not work on a CSP based subscription?

Also noticed that the cost recommendations never worked, are there any specific requirements for these to work?

Thanks for your support and keep up the great work on this tool.

Add a Consumption Usage perspective to cost recommendations

Ths requires:

  • developing a new data collection runbook for Azure Consumption Usage Details
  • add a cost perspective to the recommendations model
  • enhance recommendations runbooks with cost-related properties
  • enhance the Power BI report with a Cost perspective

LogAnalytics Custom Tables are not created

Hi, We have deployed this solution as per the documentation and added multiple Log analytics workspaces for VM/VMSS perf metrics. Runbooks are running as per schedulers, but when we check workspace for these custom tables, we dont see them and workbooks are throwing table not found errors.

AzureOptimizationVMsV1_CL
AzureOptimizationResourceContainersV1_CL
AzureOptimizationConsumptionV1_CL
AzureOptimizationAdvisorV1_CL

As result of which, recommendations exporter runbook throwing tables not found (0 records found) message.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.