Share via

Storage event trigger is running twice when a single blob is created

Marco Manfredi 0 Reputation points
2023-06-06T21:32:15.0666667+00:00

Hello,

Hope you are fine and you can help me!

I had to modify an Azure function, so i deployed a copy of it from my VSCode. I didn´t realize that I was overwriting all the App Files of the original Azure Function. For example, the host.json file was modified.

The output of my function creates a blob in a container. My issue strats here. I have a trigger (Storage event) in Azure Data Factory that executes a pipeline when the blob is created. However, after I deployed the function and lost the App Files, the trigger is detecting two events.

I suspect this is due to a temporary file being created in the container while the blob finishes. So the blob runs the trigger (eventhough it has no bytes) and then runs again.

Best Regards,

Marco.

Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
5,257 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,053 questions

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Sedat SALMAN 14,145 Reputation points MVP
    2023-06-07T03:09:46.06+00:00

    https://stackoverflow.com/questions/64970087/azure-event-trigger-fires-multiple-times-same-file

    The issue you're encountering, where the Azure Data Factory Storage Event trigger is firing twice for a single blob creation, might be due to the way the Azure Event Grid (AEG) functions. Once the event grid triggers an Azure Function for an event, it expects a response from the Azure Function within 2 minutes. If there is no response, the Event Grid will retry, leading to the perception of a second trigger event. You can mitigate this by changing the default value of retry for the Event Grid from 30 to 1. This should ensure that even for larger files that process for more than 2 minutes, a second duplicate trigger will not occur

    Another recommendation is to use the Push-Pull pattern, where the AEG delivers an event to the queue storage and then based on the needs can be processed by Azure Queue Trigger function in a concurrent manner. This pattern provides better control over event processing and avoids the potential for duplicate triggers

    https://stackoverflow.com/questions/47398333/create-a-temporary-blob-on-azure-blob-storage

    As for the possible cause of the issue being related to a temporary file being created in the container while the blob finishes, Azure Blob Storage operations are atomic. Blocks are uploaded in an uncommitted state, and the final step commits them by providing a list of their IDs. If the final step doesn't happen, you won't see the blob at all unless you explicitly ask for "uncommitted blobs". Thus, it's unlikely that a temporary blob is the cause of the double triggering you're experiencing

  2. Eyuwankg Swankg Subramanian 0 Reputation points
    2024-12-26T11:37:18.47+00:00

    Hi, were you able to solve this issue? Currently facing this problem in my pipelines and trying out different solutions.

    Let me know how to fixed it?

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.