Aggregate data in a Log Analytics workspace by using summary rules (Preview)

A summary rule lets you aggregate log data at a regular cadence and send the aggregated results to a custom log table in your Log Analytics workspace. Use summary rules to optimize your data for:

• Analysis and reports, especially over large data sets and time ranges, as required for security and incident analysis, month-over-month or annual business reports, and so on. Complex queries on a large data set often time out. It's easier and more efficient to analyze and report on cleaned and aggregated summarized data.

• Cost savings on verbose logs, which you can retain for as little or as long as you need in a cheap Basic log table, and send summarized data to an Analytics table for analysis and reports.

• Security and data privacy by removing or obfuscating privacy details in summarized shareable data and limiting access to tables with raw data.

This article describes how summary rules work and how to define and view summary rules, and provides some examples of the use and benefits of summary rules.

Here's a video that provides an overview of some of the benefits of summary rules:

How summary rules work

Summary rules perform batch processing directly in your Log Analytics workspace. The summary rule aggregates chunks of data, defined by bin size, based on a KQL query, and re-ingests the summarized results into a custom table with an Analytics log plan in your Log Analytics workspace.

You can aggregate data from any table, regardless of whether the table has an Analytics or Basic data plan. Azure Monitor creates the destination table schema based on the query you define. If the destination table already exists, Azure Monitor appends any columns required to support the query results. All destination tables also include a set of standard fields with summary rule information, including:

• _RuleName: The summary rule that generated the aggregated log entry.
• _RuleLastModifiedTime: When the rule was last modified.
• _BinSize: The aggregation interval.
• _BinStartTime: The aggregation start time.

You can configure up to 30 active rules to aggregate data from multiple tables and send the aggregated data to separate destination tables or the same table.

You can export summarized data from a custom log table to a storage account or Event Hubs for further integrations by defining a data export rule.

Example: Summarize ContainerLogsV2 data

If you're monitoring containers, you ingest a large volume of verbose logs into the ContainerLogsV2 table.

You might use this query in your summary rule to aggregate all unique log entries within 60 minutes, keeping the data that's useful for analysis and dropping data you don't need:

ContainerLogV2 | summarize Count = count() by  Computer, ContainerName, PodName, PodNamespace, LogSource, LogLevel, Message = tostring(LogMessage.Message)

Here's the raw data in the ContainerLogsV2 table:

Here's the aggregated data that the summary rule sends to the destination table:

Instead of logging hundreds of similar entries within an hour, the destination table shows the count of each unique entry, as defined in the KQL query. Set the Basic data plan on the ContainerLogsV2 table for cheap retention of the raw data, and use the summarized data in the destination table for your analysis needs.

Permissions required

Implementation considerations

• The maximum number of active rules in a workspace is 30.
• Summary rules are currently only available in the public cloud.
• The summary rule processes incoming data and can't be configured on a historical time range.
• When bin execution retries are exhausted, the bin is skipped and can't be re-executed.
• Querying a Log Analytics workspace in another tenant by using Lighthouse isn't supported.

Pricing model

There's no extra cost for Summary rules. You only pay for the query and the ingestion of results to the destination table, based on the table plan of the source table on which you run the query:

For example, the cost calculation for an hourly rule that returns 100 records per bin is:

For more information, see Azure Monitor pricing.

Create or update a summary rule

The operators you can use in summary rule your query depend on the plan of the source table in the query.

• Analytics: Supports all KQL operators and functions, except for:
  • Cross-resource queries, which use the workspaces(), app(), and resource() expressions, and cross-service queries, which use the ADX() and ARG() expressions.
  • Plugins that reshape the data schema, including bag unpack, narrow, and pivot.
• Basic: Supports all KQL operators on a single table. You can join up to five Analytics tables using the lookup operator.
• Functions: User-defined functions aren't supported. System functions provided by Microsoft are supported.

Summary rules are most beneficial in term of cost and query experiences when results count or volume are reduced significantly. For example, aiming for results volume 0.01% or less than source. Before you create a rule, experiment query in Log Analytics, and verify that the query doesn't reach or near the query API limits. Check that the query produces the intended expected results and schema. If the query is close to the query limits, consider using a smaller 'bin size' to process less data per bin. You can also modify the query to return fewer records or fields with higher volume.

When you update a query and there are fewer fields in summary results, Azure Monitor doesn't automatically remove the columns from the destination table, and you need to delete columns from your table manually.

PUT https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourcegroup}/providers/Microsoft.OperationalInsights/workspaces/{workspace}/summarylogs/{ruleName}?api-version=2023-01-01-preview
Authorization: {credential}

{
  "properties": {
      "ruleType": "User",
      "description": "My test rule",
      "ruleDefinition": {
          "query": "StorageBlobLogs | summarize count() by AccountName",
          "binSize": 30,
          "destinationTable": "MySummaryLogs_CL"
      }
  }
}

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "workspaceName": {
      "type": "String",
      "metadata": {
        "description": "The workspace name where summary rule is deployed."
      }
    },
    "summaryRuleName": {
      "type": "String",
      "metadata": {
        "description": "The summary rule name."
      }
    },
    "description": {
      "type": "String",
      "metadata": {
        "description": "A description of the rule."
      }
    },
    "location": {
      "defaultValue": "[resourceGroup().location]",
      "type": "String",
      "metadata": {
        "description": "The Location of the workspace summary rule is deployed."
      }
    },
    "ruleType": {
      "defaultValue": "User",
      "allowedValues": [
        "User"
      ],
      "type": "String",
      "metadata": {
        "description": "The summary rule type (User,System). Should be 'User' for and rule with query that you define."
      }
    },
    "query": {
      "type": "String",
      "metadata": {
      "description": "The query used in summary rules."
      }
    },
    "binSize": {
      "defaultValue": 60,
      "allowedValues": [
        20,
        30,
        60,
        120,
        180,
        360,
        720,
        1440
      ],
      "type": "Int",
      "metadata": {
        "description": "The execution interval in minutes, and the lookback time range."
      }
    },
    "destinationTable": {
      "type": "String",
      "metadata": {
        "description": "The name of the custom log table that the summary results are sent to. Name must end with '_CL'."
      }
    }
    // ----- optional -----
    // "binDelay": {
    //   "type": "Int",
    //   "metadata": {
    //     "description": "Optional - The minimum wait time in minutes before bin execution. For example, value of '10' cause bin (01:00-02:00) to be executed after 02:10."
    //   }
    // },
    // "timeSelector": {
    //   "defaultValue": "TimeGenerated",
    //   "allowedValues": [
    //     "TimeGenerated"
    //   ],
    //   "type": "String",  
    //   "metadata": {
    //     "description": "Optional - The time field to be used by the summary rule. Must be 'TimeGenerated'."
    //   }
    // },
    // "binStartTime": {
    //   "type": "String",
    //   "metadata": {
    //     "description": "Optional - The Time of initial bin. Can start at current time minus binSize, or future, and in whole hours. For example: '2024-01-01T08:00'."
    //   }
    // }
  },
  "variables": {},
  "resources": [
    {
      "type": "Microsoft.OperationalInsights/workspaces/summaryLogs",
      "apiVersion": "2023-01-01-preview",
      //"name": "[format('{0}/{1}', parameters('workspaceName'), parameters('summaryRuleName'))]",
      "name": "[concat(parameters('workspaceName'), '/', parameters('summaryRuleName'))]",
      "properties": {
        "ruleType": "[parameters('ruleType')]",
        "description": "[parameters('description')]",
        "ruleDefinition": {
          "query": "[parameters('query')]",
          "binSize": "[parameters('binSize')]",
          "destinationTable": "[parameters('destinationTable')]"
          // ----- optional -----
          //"binDelay": "[parameters('binDelay')]",
          //"timeSelector": "[parameters('timeSelector')]",
          //"binStartTime": "[parameters('binStartTime')]"
        }
      }
    }
  ]
}

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "workspaceName": {
      "value": "myworkspace"
    },
    "summaryRuleName": {
      "value": "myrulename"
    },
    "description": {
      "value": "My rule description."
    },
    "location": {
      "value": "eastus" //Log Analytics workspace region
    },
    "ruleType": {
      "value": "User"
    },
    "query": {
      "value": "StorageBlobLogs | summarize Count = count(), DurationMs98 = percentile(DurationMs, 90) by StatusCode, CallerIpAddress, OperationName"
    },
    "binSize": {
      "value": 20
    },
    "destinationTable": {
      "value": "MySummaryLogs_CL"
    }
  }
}

This table describes the summary rule parameters:

Configure the aggregation timing

By default, the summary rule creates the first aggregation shortly after the next whole hour.

The short delay Azure Monitor adds accounts for ingestion latency - or the time between when the data is created in the monitored system and the time that it becomes available for analysis in Azure Monitor. By default, this delay is between three and a half minutes to 10% of the 'bin size' value before aggregating each bin. In most cases, this delay ensures that Azure Monitor aggregates all data logged within each bin period.

For example:

• You create a summary rule with a bin size of 30 minutes at 14:44.

  The rule creates the first aggregation shortly after 15:00 - for example, at 15:04 - for data logged between 14:30 and 15:00.

• You create a summary rule with a bin size of 720 minutes (12 hours) at 14:44.

  The rule creates the first aggregation at 16:12 - 72 minutes (10% of the 720 bin size) after 13:00 - for data logged between 03:00 and 15:00.

Use the binStartTime and binDelay parameters to change the timing of the first aggregation and the delay Azure Monitor adds before each aggregation.

The next sections provide examples of the default aggregation timing and the more advanced aggregation timing options.

Use default aggregation timing

In this example, the summary rule is created at on 2023-06-07 at 14:44, and Azure Monitor adds a default delay of four minutes.

Set optional aggregation timing parameters

In this example, the summary rule is created at on 2023-06-07 at 14:44, and the rule includes these advanced configuration settings:

• binStartTime: 2023-06-08 07:00
• binDelay: 8 minutes

View summary rules

Use this GET API call to view the configuration for a specific summary rule:

GET https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourcegroup}/providers/Microsoft.OperationalInsights/workspaces/{workspace}/summarylogs/{ruleName1}?api-version=2023-01-01-preview
Authorization: {credential}

Use this GET API call to view the configuration to view the configuration of all summary rules in your Log Analytics workspace:

GET https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourcegroup}/providers/Microsoft.OperationalInsights/workspaces/{workspace}/summarylogs?api-version=2023-01-01-preview
Authorization: {credential}

Stop and restart a summary rule

You can stop a rule for a period of time - for example, if you want to verify that data is ingested to a table and you don't want to affect the summarized table and reports. When you restart the rule, Azure Monitor starts processing data from the next whole hour or based on the defined binStartTime (optional) parameter.

To stop a rule, use this POST API call:

POST https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourcegroup}/providers/Microsoft.OperationalInsights/workspaces/{workspace}/summarylogs/{ruleName}/stop?api-version=2023-01-01-preview
Authorization: {credential}

To restart the rule, use this POST API call:

POST https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourcegroup}/providers/Microsoft.OperationalInsights/workspaces/{workspace}/summarylogs/{ruleName}/start?api-version=2023-01-01-preview
Authorization: {credential}

Delete a summary rule

You can have up to 30 active summary rules in your Log Analytics workspace. If you want to create a new rule, but you already have 30 active rules, you must stop or delete an active summary rule.

To delete a rule, use this DELETE API call:

DELETE https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourcegroup}/providers/Microsoft.OperationalInsights/workspaces/{workspace}/summarylogs/{ruleName}?api-version=2023-01-01-preview
Authorization: {credential}

Monitor summary rules

To monitor summary rules, enable the Summary Logs category in the diagnostic settings of your Log Analytics workspace. Azure Monitor sends summary rule execution details, including summary rule run Start, Succeeded, and Failed information, to the LASummaryLogs table in your workspace.

We recommend you that you set up log alert rules to receive notification of bin failures, or when bin execution is nearing time-out, as shown below. Depending on the failure reason, you can either reduce the bin size to process less data on each execution, or modify the query to return fewer records or fields with higher volume.

This query returns failed runs:

LASummaryLogs | where Status == "Failed"

This query returns bin runs where the QueryDurationMs value is greater than 0.9 x 600,000 milliseconds:

LASummaryLogs | where QueryDurationMs > 0.9 * 600000

Verify data completeness

Summary rules are designed for scale, and include a retry mechanism to overcome transient service or query failures related to query limits, for example. The retry mechanism makes 10 attempts to aggregate a failed bin within eight hours, and skips a bin, if exhausted. The rule is set to isActive: false and put on hold after eight consecutive bin retries. If you enable monitor summary rules, Azure Monitor logs an event in the LASummaryLogs table in your workspace.

You can't rerun a failed bin run, but you can use the following query to view failed runs:

let startTime = datetime("2024-02-16");
let endtTime = datetime("2024-03-03");
let ruleName = "myRuleName";
let stepSize = 20m; // The stepSize value is equal to the bin size defined in the rule
LASummaryLogs
| where RuleName == ruleName
| where Status == 'Succeeded'
| make-series dcount(BinStartTime) default=0 on BinStartTime from startTime to endtTime step stepSize
| render timechart

This query renders the results as a timechart:

See the Monitor summary rules section for rule remediation options and proactive alerts.

Encrypt summary rule queries by using customer-managed keys

A KQL query can contain sensitive information in comments or in the query syntax. To encrypt summary rule queries, link a storage account to your Log Analytics workspace and use customer-managed keys.

Considerations when you work with encrypted queries:

• Linking a storage account to encrypt your queries doesn’t interrupt existing rules.
• By default, Azure Monitor stores summary rule queries in Log Analytics storage. If you have existing summary rules before you link a storage account to your Log Analytics workspace, update your summary rules so the queries to save the existing queries in the storage account.
• Queries that you save in a storage account are located in the CustomerConfigurationStoreTable table. These queries are considered service artifacts and their format might change.
• You can use the same storage account for summary rule queries, saved queries in Log Analytics, and log alerts.

Troubleshoot summary rules

This section provides tips for troubleshooting summary rules.

Summary rule destination table accidentally deleted

If you delete the destination table while the summary rule is active, the rule gets suspended and Azure Monitor sends an event to the LASummaryLogs table with a message indicating that the rule was suspended.

If you don't need the summary results in the destination table, delete the rule and table. If you need to recover summary results, follow the steps in Create or update summary rules section to recreate the table. The destination table is restored, including the data ingested before the delete, depending on the retention policy in the table.

If you don't need the summary results in the destination table, delete the rule and table. If you need the summary results, follow the steps in the Create or update summary rules section to recreate the destination table and restore all data, including the data ingested before the delete, depending on the retention policy in the table.

Query uses operators that create new columns in the destination table

The destination table schema is defined when you create or update a summary rule. If the query in the summary rule includes operators that allow output schema expansion based on incoming data - for example, if the query uses the arg_max(expression, *) function - Azure Monitor doesn't add new columns to the destination table after you create or update the summary rule, and the output data that requires these columns will be dropped. To add the new fields to the destination table, update the summary rule or add a column to your table manually.

Data in removed columns remains in the workspace based on the table's retention settings

When you remove a field in the query, the columns and data remain in the destination and based on the retention period defined on the table or workspace. If you don't need the removed in destination table, delete the columns from the table schema. If you then add columns with the same name, any data that's not older that the retention period appears again.

Related content

• Learn more about Azure Monitor Logs data plans.
• Walk through a tutorial on using KQL mode in Log Analytics.
• Access the complete reference documentation for KQL.