Deployment error when trying to enable dead-lettering on an event subscription in Azure

Hi,

We configured an event subscription (webhook call) on the system topic of an IoT hub event grid to relay events to another system. It works okay in all environments but we noticed some failed event deliveries in Prod, only a fraction of the total IoT hub events.

In order to troubleshoot we decided to enable dead-lettering on this event subscription to get more info by sending the message to a storage account container (only option).

So we modified our Terraform script to introduce the following modifications:

• create a new container in an existing storage account to store non-delivered events
• create a user assigned identity
• give this identity the Storage Blob Data Contributor role over the newly created container (not storage account)
• configure dead-lettering to send the data to the container using this user assigned identity (which is not used / assigned elsewhere, including the event grid system topic)

Running "terraform plan" works but "terraform apply" raises the following error:

│ Error: creating/updating System Topic Event Subscription (Subscription: "xxx"

│ Resource Group Name: "rg-xxx"

│ System Topic Name: "egst-xxx"

│ Event Subscription Name: "xyz"): polling after SystemTopicEventSubscriptionsCreateOrUpdate: polling failed: the Azure API returned the following error:

│

│ Status: "Failed"

│ Code: "Internal error"

│ Message: "The operation failed due to an internal server error. The initial state of the impacted resources (if any) are restored. Please try again in few minutes. If error still persists, report 7485bb7e-5fc4-4b6f-bc21-547104340cdf:12/5/2024 8:24:55 AM (UTC) to our forums for assistance or raise a support ticket ."

│ Activity Id: ""

│ ---

│ API Response:

│ ----[start]----

│ {"id":"https://management.azure.com/subscriptions/xxx/providers/Microsoft.EventGrid/locations/northeurope/operationsStatus/xxx?api-version=2022-06-15","name":"xxx","status":"Failed","error":{"code":"Internal error","message":"The operation failed due to an internal server error. The initial state of the impacted resources (if any) are restored. Please try again in few minutes. If error still persists, report 7485bb7e-5fc4-4b6f-bc21-547104340cdf:12/5/2024 8:24:55 AM (UTC) to our forums for assistance or raise a support ticket ."}}

│ -----[end]-----

│

│ with azurerm_eventgrid_system_topic_event_subscription.xyz,

Running the command later as suggested didn't solve the issue.

The container, user assigned identity and role assignment as Storage Blob Data Contributor were successfully created the first time. Only the event grid system topic update fails for a reason which is not clear (internal server error).

Can you please share more details on the error using the reference found in the error message above? I feel this has something to do with the user assigned identity not being shared / configured also on a parent resource (i.e.: event grid system topic or IoT hub) but I'd like some info to back this up if possible. Thanks for your support.