Scale Azure OpenAI for JavaScript with Azure API Management

Learn how to add enterprise-grade load balancing to your application to extend the chat app beyond the Azure OpenAI Service token and model quota limits. This approach uses Azure API Management to intelligently direct traffic between three Azure OpenAI resources.

This article requires you to deploy two separate samples:

• Chat app:
  • If you haven't deployed the chat app yet, wait until after the load balancer sample is deployed.
  • If you already deployed the chat app once, change the environment variable to support a custom endpoint for the load balancer and redeploy it again.
• Load balancer with Azure API Management.

Architecture for load balancing Azure OpenAI with Azure API Management

Because the Azure OpenAI resource has specific token and model quota limits, a chat app that uses a single Azure OpenAI resource is prone to have conversation failures because of those limits.

To use the chat app without hitting those limits, use a load-balanced solution with API Management. This solution seamlessly exposes a single endpoint from API Management to your chat app server.

The API Management resource, as an API layer, sits in front of a set of Azure OpenAI resources. The API layer applies to two scenarios: normal and throttled. During a normal scenario where token and model quota is available, the Azure OpenAI resource returns a 200 back through the API layer and backend app server.

When a resource is throttled because of quota limits, the API layer can retry a different Azure OpenAI resource immediately to fulfill the original chat app request.

Prerequisites

• Azure subscription. Create one for free

• Dev containers are available for both samples, with all dependencies required to complete this article. You can run the dev containers in GitHub Codespaces (in a browser) or locally using Visual Studio Code.

  • Codespaces (recommended)
  • Visual Studio Code

  • Only a GitHub account is required to use Codespaces

Open the Azure API Management local balancer sample app

Deploy the Azure API Management load balancer

1. To deploy the load balancer to Azure, sign in to the Azure Developer CLI (AZD):

   azd auth login
   

2. Finish the sign-in instructions.

3. Deploy the load balancer app:

   azd up
   

   Select a subscription and region for the deployment. They don't have to be the same subscription and region as the chat app.

4. Wait for the deployment to finish before you continue. This process might take up to 30 minutes.

Get the load balancer endpoint

Run the following Bash command to see the environment variables from the deployment. You need this information later.

azd env get-values | grep APIM_GATEWAY_URL

Redeploy the chat app with the load balancer endpoint

These examples are completed on the chat app sample.

Configure the TPM quota

By default, each of the Azure OpenAI instances in the load balancer is deployed with a capacity of 30,000 tokens per minute (TPM). You can use the chat app with the confidence that it's built to scale across many users without running out of quota. Change this value when:

• You get deployment capacity errors: Lower the value.
• You need higher capacity: Raise the value.

1. Use the following command to change the value:

   azd env set OPENAI_CAPACITY 50
   

2. Redeploy the load balancer:

   azd up
   

Clean up resources

When you're finished with the chat app and the load balancer, clean up the resources. The Azure resources created in this article are billed to your Azure subscription. If you don't expect to need these resources in the future, delete them to avoid incurring more charges.

Clean up the chat app resources

Return to the chat app article to clean up those resources.

• .NET
• JavaScript
• Python

Clean up the load balancer resources

Run the following Azure Developer CLI command to delete the Azure resources and remove the source code:

azd down --purge --force

The switches provide:

• purge: Deleted resources are immediately purged. You can reuse the Azure OpenAI tokens per minute.
• force: The deletion happens silently, without requiring user consent.

Clean up resources

Get help

If you have trouble deploying the Azure API Management load balancer, add your issue to the repository's Issues webpage.

Sample code

Samples used in this article include:

• JavaScript chat app with RAG
• Load Balancer with Azure API Management

Next steps

• View Azure API Management diagnostic data in Azure Monitor
• Use Azure Load Testing to load test your chat app with Azure Load Testing service.