Scraping diagnostic log files stored in azure blob storage without mounting

Andrew Yanni 0

I asked a question here about How to real-time scrape diagnostic log files stored in azure blob storage, which led me to using append blobs. This however did not solve the problem.In short, I tried scraping azure diagnostic logs as they are being written to the storage account blob containers using Promtail and Loki, but this did not work after the first few log lines.

Now I doubt this could be due to the mounting of storage accounts, and not because of not using append blobs.

Is there a way to scrape the logs being written to the azure storage accounts without mounting them at first (e.g., using kubernetes persistent volumes)

Rahul Podila 895 Reputation points Microsoft Vendor

2024-12-26T06:59:35.18+00:00

HI @Andrew Yanni
Just checking in to see if the below answer provided by @Amrinder Singh helped.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know
Paul Lee 0 Reputation points

2024-12-26T07:09:07.0233333+00:00

Scraping Azure diagnostic logs in real-time can be tricky, especially if you're relying on tools like Promtail and Loki. The challenge may indeed lie with storage mounting or file update mechanisms. Instead of mounting storage accounts directly, consider using the Azure Blob Storage SDK or Event Grid for real-time log monitoring. These tools allow you to process logs as they’re written without the need for persistent volumes. Additionally, if you’re exploring data solutions, platforms like NY Warrant Search can offer useful insights into structured records handling. This approach can streamline log scraping and avoid unnecessary storage dependencies.
Rahul Podila 895 Reputation points Microsoft Vendor

2024-12-26T09:59:44.8333333+00:00

Hi @Paul Lee
Thanks for the clarification! It sounds like you're running into some challenges with real-time log scraping from Azure Blob Storage, especially with mounting the storage in Kubernetes. I can understand how that might cause issues, and I think there's a better way to handle this without the mounting step.

Instead of mounting the Blob Storage, I recommend using Azure Event Grid. Event Grid can send you notifications whenever a new log file is uploaded or updated in your Blob Storage. This way, you won’t have to rely on mounting the storage in Kubernetes at all.

First, you’ll need to configure Event Grid to listen for events like Blob Created or Blob Modified. These events will notify you when a new log file is added or updated. Once an event is triggered (e.g., when a new log file is created), you can use an Azure Function or an HTTP endpoint to receive that notification. This handler will know exactly when new logs are available.

When the handler receives the notification, you can tell Promtail to fetch and scrape the new log files from Azure Blob Storage and send them to Grafana Loki for monitoring.

This approach means you won’t need to mount Blob Storage, and you’ll get logs in real-time as soon as they’re written. It should also be much more scalable and reliable than relying on volume mounting.

If you have any further queries, do let us know.
Rahul Podila 895 Reputation points Microsoft Vendor

2024-12-27T01:17:53.5966667+00:00

Hi @Paul Lee
If you had a chance to see my comment to your question. If it was helpful, please click "Upvote" on my post let us know Thank you...!

1 answer

Amrinder Singh 5,555 Reputation points Microsoft Employee

2024-12-09T14:31:55.66+00:00

Hi Andrew Yanni - Thanks for reaching out over Q&A Forum.

Just to clarify, are you referring to Diagnostic logging or change feed logs?

If change feed, there is retention you configure (min 1 day)

If it is only deletion of the append blobs, you can try leveraging LCM as it supports deletion of the Append Blobs.

https://zcusa.951200.xyz/en-us/azure/storage/blobs/lifecycle-management-overview#rule-actions

delete2,3 Supported for blockBlob and appendBlob Supported Supported

delete2,3 Supported for blockBlob and appendBlob Supported Supported
Please sign in to rate this answer.
Andrew Yanni 0 Reputation points

2024-12-11T09:31:19.32+00:00

Hi Amrinder, I am referring to Diagnostic logging.

Andrew Yanni 0 Reputation points

2024-12-11T09:32:02.49+00:00

Hi Amrinder, I think you misunderstood what I meant. I did not want to delete the log lines, but rather stream them to Grafana Loki using one of the scrape_config options of Grafana Promtail that specifies a static target to scrape.

In some of my attempts I had the storage containers (names insights-logs-xyz as you suggested) get mounted as kubernetes volumes and mount the volumes to the Promtail container. This initially worked for the first few log lines, but then stopped due to mounting/streaming issues.

Amrinder Singh 5,555 Reputation points Microsoft Employee

2024-12-11T14:48:52.1166667+00:00

Hi Andrew Yanni - So in the case of diagnostic logging, the container name shall be looking like insights-logs-storageread, insights-logs-storagewrite or insights-logs-storagedelete depending upon the logs category.

You can have a LCM rule in place with prefix match as that of container name and have it tested further.

Hope this information helps! please let us know if you have any further queries. I’m happy to assist you further.

Please "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Scraping diagnostic log files stored in azure blob storage without mounting

1 answer

Your answer