Concurrent Device Connections and Telemetry on Azure IoT Hub

Details:

• SDK Versions:
  • azure-iot-device : 2.14.0
    • azure-iot-hub : 2.6.1
      • asyncio : 3.4.3
        • Python Version : 3.10.12
        • Azure IoTHub Details:
          • Tier: S1

Description of Issue: I have developed a Python script using the Azure IoT SDK to connect 670 devices to Azure IoT Hub. The script is designed to connect each device asynchronously, update the device twin, and send telemetry data in parallel.

However, I am facing a connectivity limitation when running the script for all 670 devices simultaneously:

• After 337 devices successfully connect, update their twins, and start sending telemetry, the remaining devices are unable to establish a connection.
• Logs indicate a new connection error for devices beyond the initial 337 connections, with the following details:Details:
  • SDK Versions:
    • azure-iot-device : 2.14.0
    • azure-iot-hub : 2.6.1
    • asyncio : 3.4.3
    • Python Version : 3.10.12
  • Azure IoTHub Details:
    • Tier: S1
  Description of Issue:
  I have developed a Python script using the Azure IoT SDK to connect 670 devices to Azure IoT Hub. The script is designed to connect each device asynchronously, update the device twin, and send telemetry data in parallel.   However, I am facing a connectivity limitation when running the script for all 670 devices simultaneously:
  • After 337 devices successfully connect, update their twins, and start sending telemetry, the remaining devices are unable to establish a connection.
  • Logs indicate a new connection error for devices beyond the initial 337 connections, with the following details:
• pipeline_stages_mqtt.py:83 - MQTTTransportStage(ConnectOperation): Connection watchdog expired. Cancelling op mqtt_transport.py:441 - disconnecting MQTT client async_adapter.py:72 - Callback completed with error Transport timeout on connection operation async_adapter.py:73 - ['azure.iot.device.common.pipeline.pipeline_exceptions.OperationTimeout: Transport timeout on connection operation\n'] mqtt_transport.py:150 - Creating client for connecting using MQTT over TCP

In the exception, the error message received is: Error: Could not complete operation before timeout.

Interestingly, when I divide the workload into two batches of 335 devices each, running them in separate instances of a python script from the same IP address, all connections are successful, and each device updates its twin and sends telemetry as expected. Only same python process is not able to connect to all devices.