Azure IoT Operations Dataflow sync very slow

Neil Smith 20

Dataflows stay in the provisioning state 'Accepted' for many hours. Sometimes they eventually go to 'Failed'. Is intermittent, sometimes they synchronize fully in about 30 seconds

AshokPeddakotla-MSFT 35,576 Reputation points

2024-12-11T14:30:14.6+00:00

Neil Smith Greetings & Welcome to Microsoft Q&A forum!

Dataflows stay in the provisioning state 'Accepted' for many hours. Sometimes they eventually go to 'Failed'. Is intermittent, sometimes they synchronize fully in about 30 seconds

Do you also see any particular error message when state goes to 'Failed'?
AshokPeddakotla-MSFT 35,576 Reputation points

2024-12-16T11:19:31.3633333+00:00

Neil Smith Just wanted to check if the answer provided below was helpful? If it answers your query, please do click Accept Answer and Yes for the answer, as it might be beneficial to other community members reading this thread.

And, if you have any further query do let us know.

Accepted answer

Sander van de Velde | MVP 33,956 Reputation points MVP

2024-12-11T17:14:18.17+00:00

Hello @Neil Smith,

welcome to this moderated Azure community forum.

I expect the Azure IoT Operations deployment is running successfully.

What kind of data flow (which source and destination, and possible transformations) is deployed?

Deploying a data flow takes some time (because these need to be merged on the edge with the current situation).

I just finished this blog post about deploying data flows using Bicep scripts.

There you can see how to deploy a basic data flow from local mqtt broker to local mqtt broker via Bicep.

I can also recommend adding transformations step by step so you know for sure what is working and what is not.

Please share your experience with more details so other community members understand the context better.

If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.
Please sign in to rate this answer.
Neil Smith 20 Reputation points

2024-12-11T17:56:44.4466667+00:00

I didn't notice any error message and didn't go delving in to container logs, I just saw state as Failed, I don't have any failed at the moment so can't check for error messages.

Thanks for your blog post looks very interesting, perhaps I'll make them in Bicep in future. I made a simple dataflow, the source is the default MQTT instance, I then add a couple of properties to the json and the destination is an event hub on azure. I currently make them using the web front end, iotopertions.azure.com. I just had a fairly simple modification stuck in the Accepted state for 6 hours before changing to Succeeded

Sander van de Velde | MVP 33,956 Reputation points MVP

2024-12-11T18:24:21.04+00:00

Hello @Niel Smith,

my experience is that a data flow update, even with transformations that do not make sense, take less than 30 seconds (at least less than a minute).

You could experiment with the data flow by adding or deleting some of the transformations. Perhaps you can gain and share more insights this way.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

1 additional answer

Ernesto Guimaraes 0 Reputation points Microsoft Employee

2024-12-27T14:24:50.1733333+00:00
Go to your Aio Cluster and delete the POD aio-dataflow-operator-0

This action make my service to restart and execute the provisioning state.

kubectl delete pod aio-dataflow-operator-0 -n azure-iot-operations
Please sign in to rate this answer.
Sander van de Velde | MVP 33,956 Reputation points MVP

2024-12-27T17:38:30.9933333+00:00

Yes, eventually I ran into this behavior and deleting the aio-dataflow-operator-0 (and potentially aio-dataflow-default-0 too) is a workaround that works. These pods are automatically restarting.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Azure IoT Operations Dataflow sync very slow

1 additional answer

Your answer