Storge Account - Life Cycle Management

Santhi Dhanuskodi 305

Hi,

I want to add a rule to remove blobs which are more than 30 days old in a storage container.

My files are present in <cotnainername>/Backup/<Datetime>/files.csv......

I want to remove only files under this backup folder. I dont want to delete anything other than this backup folder.

How I can create a rule? How do I check if a rule has been run and its logs?

I saw event subscription, but it is asking to add the event logs into someother azure resources.

Should I then purchase any of these services and manage them? Is there an inbuilt logs within storage account that we can see? If there is a failure of rule, how do we go about it?

User's image

Nehruji R 7,811 Reputation points Microsoft Vendor

2024-10-01T07:51:34.6866667+00:00
Hello Santhi Dhanuskodi,

Greetings! Welcome to Microsoft Q&A Platform.

To delete older files from each folder in Azure Blob Storage, you have a couple of options:

#1. Blob Lifecycle Management (Recommended): Azure Storage provides a feature called Blob Lifecycle Management. With this, you can define policies to automatically delete blobs based on criteria such as their age (e.g., older than 30 days). Go to your Azure Storage Account.Navigate to Blob Service > Lifecycle Management. Define a policy to delete blobs older than your desired threshold (e.g., 14 days). Azure Storage will handle the deletion for you. refer -https://zcusa.951200.xyz/en-us/azure/storage/blobs/lifecycle-management-policy-configure?tabs=azure-portal, https://zcusa.951200.xyz/en-us/azure/storage/blobs/lifecycle-management-overview#rule-actions

#2. Azure Functions: You can create an Azure Function that periodically checks each folder and deletes files older than a specified duration. Here’s a high-level approach: Create a Timer Trigger function. Fetch the list of blobs from the corresponding Blob Container. Check the LastModified property of each blob. Delete the blobs that don’t meet your criteria (e.g., older than 14 days).

Remember to test any deletion process on a test folder first to avoid accidental data loss. Adjust the path and days as needed for each folder.

You can use Azure Monitor and Azure Storage Analytics to track and log activities.

Create a Log Analytics workspace and Set up diagnostic settings to send logs to this Workspace.

Use Log Analytics to query and analyze the logs.

Handling Failures: If a rule fails, you can troubleshoot using the logs collected in Azure Monitor.

Note: If you need detailed event logs, you might need to set up event subscriptions and route logs to other Azure services like Event Grid or Log Analytics and this may incur additional costs.

Similar thread for reference - https://zcusa.951200.xyz/en-us/answers/questions/1299568/how-to-check-life-cycle-management-rules-actions

Hope this answer helps! Please let us know if you have any further queries. I’m happy to assist you further.

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.
Santhi Dhanuskodi 305 Reputation points

2024-10-01T08:58:46.5+00:00

My questions are towards lifecycle management.

How I can create a rule? How do I check if a rule has been run and its logs?

There is something like event subscription, but it is asking to add the event logs into someother azure resources.

Should I then purchase any of these services and manage them? Is there an inbuilt logs within storage account that we can see? If there is a failure of rule, how do we go about it?

how can i know that how many files are deleted in this process? and wat are those file names etc?

Can it log by default to azure diagnostics?
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Nehruji R 7,811 Reputation points Microsoft Vendor

2024-10-03T09:19:32.76+00:00

Hi, we hope the below provided answer helped if yes please accept the answer else, please let us know if you have any further queries. I’m happy to assist you further.

1 answer

Sumarigo-MSFT 46,286 Reputation points Microsoft Employee

2024-10-01T11:20:11.83+00:00
@Santhi Dhanuskodi Welcome to Microsoft Q&A Forum, Thank you for posting your query here!

As I assume you need to you have apply policy only for "Backup" folder which is under container and policies should not scan/apply for other folder inside backup folder i.e Datetime am I correct?

<cotnainername>/Backup/<Datetime>/files.csv......

Note : We do not have features to exclude few folders that are inside backup fokder

I want to remove only files under this backup folder. I dont want to delete anything other than this backup folder. This option isn't possible for now.

Configure a lifecycle management policy

Question: How I can create a rule? How do I check if a rule has been run and its logs?
Azure Blob Storage frequently asked questions : This article provides a list of frequently asked questions (FAQ) for Azure Blob Storage |Lifecycle management policies

To check if a blob lifecycle management policy has been applied, navigate to your Azure storage account in the portal, go to "Data management" then "Lifecycle Management", where you can view the list of configured rules and their status; you can also monitor policy execution by checking the storage account's metrics and logs to see if blobs are being transitioned or deleted according to the policy rules.

There is a similar thread discussion : How to check Life Cycle management rules actions : https://zcusa.951200.xyz/en-us/answers/questions/1299568/how-to-check-life-cycle-management-rules-actions

Question : I saw event subscription, but it is asking to add the event logs into someother azure resources.
The LifecyclePolicyCompleted event is generated when the actions defined by a lifecycle management policy are performed. Refer - Optimize costs by automatically managing the data lifecycle - Azure Blob Storage | Microsoft Learn

This article shares steps to subscribe to these events and help you track how much data moved, deleted or archived.

How to view deleted files of Azure Storage Lifecycle management? : https://stackoverflow.com/questions/68541332/how-to-view-deleted-files-of-azure-storage-lifecycle-management

The blog talks about how you can leverage the existing metrics and diagnostic logging to monitor or track the execution of lifecycle management policies. Monitoring Life Cycle Management Policy Runs

You don't necessarily need to purchase additional services to manage logs. Azure Blob Storage has inbuilt logging capabilities. You can enable diagnostic settings for your storage account and save the logs to the Log Analytics Workspaces on Azure. The logs are saved in a blob container named $logs in your storage account.
Enable and manage Azure Storage Analytics logs (classic)

You don't necessarily need to purchase additional services to manage logs. Azure Blob Storage has inbuilt logging capabilities. You can enable diagnostic settings for your storage account and save the logs to the Log Analytics Workspaces on Azure. The logs are saved in a blob container named$logsin your storage account, Refer to more information here

If a rule fails, you can review the Azure Blob Storage logs and check for any error messages. Ensure that the lifecycle rule is properly applied to the storage account and that the storage account has the necessary permissions to perform the specified actions. Subscribe to Azure Storage Blob Lifecycle Policy Events

To know how many files are deleted and their names, you can subscribe to the LifecyclePolicyCompleted event, which provides details about the data moved, deleted, or archived

By default, you can log to Azure diagnostics by enabling the diagnostic setting for your storage account and saving the logs to the Log Analytics Workspaces

Additional information :

Rule actions: Actions are applied to the filtered blobs when the run condition is met. Lifecycle management supports tiering and deletion of current versions, previous versions, and blob snapshots. Define at least one action for each rule.

Note: If you define more than one action on the same blob, lifecycle management applies the least expensive action to the blob. For example, action delete is cheaper than action tierToArchive. Action tierToArchive is cheaper than action tierToCool.
Lifecycle policy runs: When you configure or edit a lifecycle policy., it can take up to 24 hours for changes to go into effect and for the first execution to start. The time taken for policy actions to complete depends on the number of blobs evaluated and operated on. For more information refer to this article.

Is there a way to identify the time at which the policy will be executing?

Unfortunately, there's no way to track the time at which the policy will be executing, as it's a background scheduling process. However, the platform will run the policy once per day.

Please let us know if you have any further queries. I’m happy to assist you further.

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.
Please sign in to rate this answer.
Santhi Dhanuskodi 305 Reputation points

2024-10-03T09:24:25.3033333+00:00

I can see the details in Stroage Account Metrics

Also I can see the details in StorageBlobLogs by running the query

I have not subscribed to the event LifecyclePolicyCompleted. But I can still see the details in the above ways. How LifecyclePolicyCompleted event will generate logs compared to the above logs? Wat additional details it can provide? Is it mandatory?

Santhi Dhanuskodi 305 Reputation points

2024-10-03T10:02:15.7333333+00:00

We can see logs in two places 1. Metrics

without subscribing to the event.

How "LifeCycleEventCompleted' adds value to these logs? Wat is different from the logs generated out of the event vs above logs? How these logs are generated without subscribing to the event?

Santhi Dhanuskodi 305 Reputation points

2024-10-03T10:14:19.5+00:00

I havent subscribed to 'LifecyclePolicyCompleted' event, but still I can get to see the logs in Metrics

How these logs are different from 'LifecyclePolicyCompleted' event logs? what the event logs offer additionally? How event logs will look like? Any sample?

How come these logs are generated without event subscription?

Sumarigo-MSFT 46,286 Reputation points Microsoft Employee

2024-10-04T07:36:00.44+00:00

@Santhi Dhanuskodi Can you please share the details asked in Private message?

Santhi Dhanuskodi 305 Reputation points

2024-10-04T13:02:52.5166667+00:00

I have responded to your question. I am asking technically about the logs generation without and with event subscription. Is it not possible to get it from MS?

KarishmaTiwari-MSFT 20,052 Reputation points Microsoft Employee

2024-10-04T15:24:24.8+00:00

Santhi Dhanuskodi Thanks for the information. I am looking into it and will get back shortly, as I have an update.

KarishmaTiwari-MSFT 20,052 Reputation points Microsoft Employee

2024-10-04T21:21:13.88+00:00

Santhi Dhanuskodi

Q: How Metrics logs are different from 'LifecyclePolicyCompleted' event logs?
#Metrics are aggregated data points that provide insights into the performance and usage of your resources over time. They are typically available in a more granular form and can include statistics like blob count, transaction count, and capacity metrics.

Metrics are useful for monitoring the overall health and performance of your storage account. These logs are automatically collected and available without any additional configuration.

#Events are notifications about specific occurrences within Azure resources. The LifecyclePolicyCompleted event specifically indicates that a lifecycle policy operation (like deletion or moving to a cooler storage tier) has been successfully completed.

Events are useful for triggering workflows or automation, such as running a function app when a lifecycle policy is applied or completed.

Q: what the event logs offer additionally? How event logs will look like? Any sample?
Events provide real-time notifications when a specific action occurs, allowing you to respond immediately.

Events usually contain additional context such as the specific blob affected, the operation that was completed, and a timestamp. This can help in auditing and tracking changes to your blobs.

The following json shows an example LifecyclePolicyCompleted event.

{ "topic": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/contosoresourcegroup/providers/Microsoft.Storage/storageAccounts/contosostorageaccount", "subject": "BlobDataManagement/LifeCycleManagement/SummaryReport", "eventType": "Microsoft.Storage.LifecyclePolicyCompleted", "eventTime": "2022-05-26T00:00:40.1880331", "id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "data": { "scheduleTime": "2022/05/24 22:57:29.3260160", "deleteSummary": { "totalObjectsCount": 16, "successCount": 14, "errorList": "" }, "tierToCoolSummary": { "totalObjectsCount": 0, "successCount": 0, "errorList": "" }, "tierToArchiveSummary": { "totalObjectsCount": 0, "successCount": 0, "errorList": "" } }, "dataVersion": "1", "metadataVersion": "1" }

Q: How come these logs are generated without event subscription?
Azure automatically collects and retains certain metrics for storage accounts without requiring explicit subscriptions. These metrics are part of the Azure monitoring infrastructure and are available by default.

The data reflected in the Metrics tab provides insights based on system-wide activities and does not rely on event subscriptions, allowing users to access important usage information right away.

Let me know if you have any other questions.

Sumarigo-MSFT 46,286 Reputation points Microsoft Employee

2024-10-07T06:56:02.3766667+00:00

@Santhi Dhanuskodi Adding more information to the above repsonse!

The above LCM rule should help in achieving your ask. More info about prefixMatch is here.

When you delete any file in the container and also those trancation will be captured in the Metric before deleting any file in the blob container and soft delete is enabled you will get this alert

Analyze metrics with Azure Monitor metrics explorer

Additional information:

LifecyclePolicyCompleted event in the storage account's activity log. The LifecyclePolicyCompleted event is logged when a lifecycle management policy has completed its execution.

To view the activity log for your storage account, follow these steps:

Go to the Azure portal and navigate to your storage account

Under Monitoring, select Activity log.

In the Filter pane, select the Subscription, Resource group, and Resource type for your storage account

In the Search box, enter LifecyclePolicyCompleted and press Enter.

You should see a list of events that match your search criteria. You can click on an event to view its details, including the time it was logged and the status of the lifecycle management policy.

If you are not seeing any LifecyclePolicyCompleted events in the activity log, it is possible that the lifecycle management policy has not been executed yet. If the policy is not executing please let me know and we can investigate with you directly.

When you inform : LifecyclePolicyCompleted' event, but still I can get to see the logs in Metrics

Still, I can't be able to see the LifecyclePolicyCompleted event log in my activity log

So, you can use the below KQL query to check the lifecycle management policy change to another tier.

StorageBlobLogs |where UserAgentHeader contains "ObjectLifeCycleScanner" and OperationName contains "setblobtier" |project AccountName,UserAgentHeader,OperationName,AccessTier,TimeGenerated,ServiceType,Uri

The above query retrieves logs from Azure Storage where the ObjectLifeCycleScanner changed a blob's access tier (setblobtier operation), showing details like account name, access tier, and operation time. It's used for monitoring lifecycle management activities in Azure Blob Storage.

You can also refer this MS-Document to subscribe to Azure Storage Blob Lifecycle Policy Events

Please let us know if you have any further queries. I’m happy to assist you further.

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Storge Account - Life Cycle Management

1 answer

Your answer