How to set up capture to AzureStorage from EventHub Avro type files

Mirko Boruc (MKBU) 0 Reputation points
2024-12-01T18:33:23.4133333+00:00

Hi, I am trying to set up Avro files capture for EventHub to be stored on Azure Storage. I have a container, granted access to it as a Storage Contributor to aforementioned EventHub. When saving, I am getting all the time

obraz

Not sure if anything is wrong or there is an issue in Azure.

Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
3,292 questions
Azure Event Hubs
Azure Event Hubs
An Azure real-time data ingestion service.
663 questions
{count} votes

2 answers

Sort by: Most helpful
  1. Luis Arias 7,766 Reputation points
    2024-12-01T19:50:37.82+00:00

    Hello Mirko,

    Based on the error you need to add the data plane access to Azure Storage resources. For accessing Blobs only contributor access isn't enough so you need specific Azure RBAC roles:

    • Storage Blob Data Contributor: Grants full access to blob data (read, write, delete).

    Steps to Resolve

    • Assign the appropriate role via Azure Portal:
    • Go to Storage Account > Access Control (IAM) > Add Role Assignment.
      • Assign the relevant role (e.g., Storage Blob Data Contributor) to your user.
      • Alternatively, assign via Azure CLI:BashCopy
            az role assignment create \
            --role "Storage Blob Data Contributor" \
            --assignee <YourUserPrincipalName> \
            --scope /subscriptions/<SubscriptionId>/resourceGroups/<ResourceGroup>/providers/Microsoft.Storage/storageAccounts/<StorageAccountName>
      

    Reauthenticate in Azure Storage Explorer after role assignment to ensure access.

    References

    If this resolves your issue, please accept the answer.

    Luis

    0 comments No comments

  2. Sumarigo-MSFT 47,371 Reputation points Microsoft Employee
    2024-12-05T09:32:07.95+00:00

    @Mirko, Welcome to Microsoft Q&A Forum, Thank you for posting your query!

    Adding more information to the above response!

    Based on the error message: Please refer to the suggestion mentioned in this GitHub thread

    To resolve the issue, you need to grant the user assigned managed identity with Storage Blob Data Owner permissions on the storage account. You can do this by adding a role assignment for the managed identity on the storage account with the required permissions. Specifically, you need to add the Microsoft.Storage/storageAccounts/blobServices/containers/write and Microsoft.Storage/storageAccounts/blobServices/containers/blobs/write permissions to the role assignment.

    For more detials, refer to Capture events through Azure Event Hubs in Azure Blob Storage or Azure Data Lake Storage - Azure Storage account as a destination

    This article lists some examples of role assignment conditions for controlling access to Azure Blob Storage. Example Azure role assignment conditions for Blob Storage
    Security considerations for Azure role assignment conditions in Azure Blob Storage
    Azure Event Hubs enables you to automatically capture the streaming data in Event Hubs in an Azure Blob storage or Azure Data Lake Storage Gen 1 or Gen 2 account of your choice. Setting up Capture is fast, there are no administrative costs to run it, and it scales automatically with Event Hubs throughput units. The files produced by Event Hubs Capture have the Avro schema. You can process these Avro files with Azure Data Lake Analytics using U-SQL

    If you encounter issues, ensure that the destination storage account is in the same subscription as the event hub when not using managed identity for authentication. Event Hubs doesn't support capturing events in a premium storage account

    0 comments No comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.