Past Saturday (July 27) in the morning, our software stopped working because of an azure service bus error.
Cannot allocate more handles. The maximum number of handles is 4999
The sender is a dotnet C# api that uses ServiceBusClient / ServiceBusSender to send the message. ServiceBusClient is retrieved using Dependency Injection.
The servicebus sender is created in the constructor and then the message is send in the [HttpPost] function.
I have now moved the creation of the sender to the [HttpPost] function and call DisposeAsync() after sending the message. This made the service healthy again, although deploying an update causes the appservice to restart which would likely have fixed it too. So I am unsure if this really fixes anything.
The servicebusclient is added to dependency injection using the following code.
var serviceBusNamespace = builder.Configuration["Settings:ServiceBusNamespace"];
builder.Services.AddAzureClients(builder =>
{
builder.AddServiceBusClientWithNamespace(serviceBusNamespace);
builder.UseCredential(new DefaultAzureCredential(true));
});
Metrics
- When looking at the 'ActiveConnections' metric, aggregation 'Max'. I always see only 1 connection active. Max Connections Closed/Opened metrics always show 0 or 1.
- There was a peak of 124 incoming requests at july 26 5:15PM, but after that it was at most 5 until the service went down the following morning.
I can not find a metric that shows us coming near the 5000 handles.
Architecture
Sender: Dotnet api running on Premium v3 plan (P1v3), single instance
Using Azure.Messaging.ServiceBus 7.17.5. (I will update this to 7.18.0.)
Receiver: Azure functions app running on the same P1v3 plan
Using Microsoft.Azure.Functions.Worker.Extensions.ServiceBus 5.20.0
ServiceBus: Basic, containing 2 queues (both use the same sender/receiver apps)
Recent changes
In the week before this issue happened, we added a healthcheck. All this does is creating a receiver calling PeekMessageAsync().
After the error occured I also added DisposeAsync to the healthcheck.
Questions
- Am I supposed to call DisposeAsync after sending/peeking a message? (The example code doesn't show this).
- How can I see how many handles I am currently using?
- What can I do to avoid this in the future?