Share via

Get started with the Azure Optimization Engine

  • Article

The Azure Optimization Engine (AOE) is an extensible solution designed to generate optimization recommendations for your Azure environment. See it like a fully customizable Azure Advisor.


Prerequisites

Here are the requirements for deploying AOE:

  • A supported Azure subscription (see the FAQ)
  • A user account with Owner permissions over the chosen subscription, so that the Automation Managed Identity is granted the required privileges over the subscription (Reader) and deployment resource group (Storage Blob Data Contributor)
  • Azure PowerShell 9.0.0+
  • Optional, for Identity and Azure role-based access control (RBAC) (RBAC_ governance) Microsoft.Graph.Authentication and Microsoft.Graph.Identity.DirectoryManagement PowerShell modules (version 2.4.0+).
  • Optional, for Identity and Azure RBAC governance. A user account is needed with at least Privileged Role Administrator permissions over the Microsoft Entra tenant, so that the Managed Identity is granted the required privileges over Microsoft Entra ID (Global Reader).
  • Optional, for Azure commitments insights. A user account is needed with administrative privileges over the Enterprise Agreement (Enterprise Enrollment Administrator) or the Microsoft Customer Agreement (Billing Profile Owner). The account is needed so that the Managed Identity is granted the required privileges over your consumption agreement.

During deployment, you're asked several questions. You must plan for the following items:

  • Determine whether you're going to reuse an existing Log Analytics Workspace or if you want to create a new one.

    Important

    You should ideally reuse a workspace where you have VMs already sending performance metrics (Perf table), otherwise you will not fully leverage the augmented right-size recommendations capability. If this is not possible/desired for some reason, you can still manage to use multiple workspaces (see Configuring workspaces).

  • An Azure subscription to deploy the solution is needed. If you're reusing a Log Analytics workspace, you must deploy into the same subscription the workspace is in.
  • A unique name prefix is needed for the Azure resources that get created. If you have specific naming requirements, you can also choose resource names during deployment.
  • Azure region
  • Optional, for Azure commitments insights. An Enterprise Agreement Billing Account ID (EA/Microsoft Customer Agreement (MCA) customers) and the Billing Profile IDs (MCA customers) are needed.

Why an optimization engine?

The Azure Optimization Engine (AOE) was initially developed to augment Virtual Machine right-size recommendations coming from Azure Advisor with more metrics and properties. You can read the blog series dedicated to the idea at Augmenting Azure Advisor cost recommendations for automated continuous optimization – Part 1. It evolved to a generic framework for Well-Architected Framework-inspired optimizations of all kinds, developed by the community. Besides the recommendations generated by Azure Advisor, AOE includes several custom recommendations, mostly from the Cost pillar, and allows for the rapid development of new ones. AOE complements Azure Advisor and other first-party Azure services with more optimization insights and allows for full customization.


Benefits

Besides collecting all Azure Advisor recommendations, AOE includes other custom recommendations that you can tailor to your needs, such as:

  • Cost
    • Augmented Advisor virtual machine (VM) right-sizing cost recommendations, with fit score based on virtual machine guest OS metrics (collected by Azure Monitor agents) and Azure properties
    • Underutilized Azure Virtual Machine Scale Sets, premium SSD disks, App Service plans, and Azure SQL databases (DTU-based SKUs only)
    • Orphaned disks and public IPs
    • Standard load balancers or application gateways without backend pool
    • VMs deallocated since a long time ago (forgotten VMs)
    • Storage accounts without retention policy in place
    • App Service plans without any application
    • Stopped (not deallocated) virtual machines
  • High availability
    • Virtual machine high availability (availability zones count, availability set, managed disks, storage account distribution when using unmanaged disks)
    • Virtual Machine Scale Sets high availability (availability zones count, managed disks)
    • Availability sets structure (fault/update domains count)
  • Performance
    • Virtual Machine Scale Sets constrained by lack of compute resources
    • SQL databases constrained by lack of resources (DTU-based SKUs only)
    • App Service plans constrained by lack of compute resources
  • Security
    • Service principal credentials/certificates without expiration date
    • NSG rules referring to empty/unexisting subnets, orphan/removed NICs, and orphan/removed public IPs
  • Operational excellence
    • Basic load balancers without backend pool
    • Service principal credentials/certificates expired or about to expire
    • Subscriptions and management groups close to the maximum limit of Azure RBAC assignments
    • Subscriptions close to the maximum limit of resource groups
    • Empty subnets and subnets with low free IP space or with too much IP space wasted
    • Orphaned NICs

In addition to the custom recommendations generated every week, AOE includes the following Azure workbooks that provide deep insights about:


What's included

AOE includes the following resources:

  • Storage account to hold all raw data exports
  • Log Analytics workspace where data is ingested and processed to generate recommendations and insights
  • Azure Automation instance to manage data ingestion and recommendations generation logic
  • Azure SQL database to hold up to one year of recommendations history, ingestion control data, and recommendations suppression records
  • The following Azure workbooks, sitting on top of the Log Analytics data:
    • Benefits simulation
    • Benefits usage
    • Block blob storage usage
    • Costs growing
    • Identities and roles
    • Policy compliance
    • Recommendations
    • Reservations potential
    • Reservations usage
    • Resources inventory
    • Savings plans usage
  • A Power BI report with the most recent recommendations

After deployment and initial ingestion and recommendation-generation automation completes, typically after three hours, you can report on the data with the help of Azure workbooks or Power BI.


Deploy the AOE

The simplest, quickest, and recommended method for installing AOE is by using the Azure Cloud Shell (PowerShell). You just have to follow these steps:

  1. Open Azure Cloud Shell (PowerShell)
  2. Run git clone https://github.com/microsoft/finops-toolkit.git
  3. Run cd finops-toolkit/src/optimization-engine
  4. Run git checkout main
  5. (optional) Run Install-Module Microsoft.Graph.Authentication,Microsoft.Graph.Identity.DirectoryManagement - this step is required to grant the Global Reader role to the Automation Managed Identity in Microsoft Entra ID, used by Identity and RBAC governance features.
  6. Run ./Deploy-AzureOptimizationEngine.ps1
  7. Input your deployment options and let the deployment finish (it takes less than five minutes)

If the deployment fails for some reason, you can repeat it, as it's idempotent. The same if you want to upgrade a previous deployment with the latest version of the repo. You just have to keep the same deployment options. The deployment script persists your previous deployment options and lets you reuse it.

If you don't want to use Azure Cloud Shell and prefer instead to run the deployment from your workstation's file system, you must first install Azure PowerShell and also the Microsoft.Graph modules.

Optionally, you can specify the set of tags you want to assign to your AOE resources, by using the ResourceTags input parameter. For example:

$tags = @{"Service"="aoe";"Environment"="Demo"}
.\Deploy-AzureOptimizationEngine.ps1 -ResourceTags $tags

Start using the AOE

After you deploy AOE, there are several ways for you to get started (you have to wait at least three hours before seeing data):

  1. Explore the several available Azure Workbooks, starting with the Recommendations one. AOE Workbooks are available from within the Log Analytics workspace chosen during installation (check the Workbooks window inside the workspace). For more information, see Reports.
  2. Open the built-in Power BI report to get deeper insights about recommendations and customize it to your needs. For more information, see Reports.
  3. Customize AOE by widening the scope of the engine or adjusting thresholds to your needs. You can do it right after deployment. For all the available customization details, check Customizations.
  4. For richer virtual machine right-size recommendations, you can add your machines' performance logs to the scope of AOE. Check Configuring workspaces.

Every week at the same time, AOE recommendations are updated according to the current state of your environment.


Related content

Related FinOps capabilities:

Related products:

Related solutions: