Backend Server Health Status Conflicting with Health Probe Logs in Application Gateway

Hi Arunkumar Elango

Greetings!

Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.

Message: Application Gateway could not connect to the backend. Check that the backend responds on the port used for the probe. Also check whether any NSG/UDR/Firewall is blocking access to the Ip and port of this backend.

Cause: After the DNS resolution phase, Application Gateway tries to connect to the backend server on the TCP port configured in the HTTP settings. If Application Gateway can't establish a TCP session on the port specified, the probe is marked as Unhealthy with this message.

Solution: If you receive this error, follow these steps:

1. Check whether you can connect to the backend server on the port mentioned in the HTTP settings by using a browser or PowerShell. For example, run the following command: Test-NetConnection -ComputerName www.bing.com -Port 443.
2. If the port mentioned isn't the desired port, enter the correct port number for Application Gateway to connect to the backend server.
3. If you can't connect on the port from your local machine as well, then: a. Check the network security group (NSG) settings of the backend server's network adapter and subnet and whether inbound connections to the configured port are allowed. If they aren't, create a new rule to allow the connections. To learn how to create NSG rules, see the documentation page. b. Check whether the NSG settings of the Application Gateway subnet allow outbound public and private traffic, so that a connection can be made. Check the document page that's provided in step 3a to learn more about how to create NSG rules. Azure PowerShellCopy

              $vnet = Get-AzVirtualNetwork -Name "vnetName" -ResourceGroupName "rgName"
           Get-AzVirtualNetworkSubnetConfig -Name appGwSubnet -VirtualNetwork $vnet
   

   c. Check the user-defined routes (UDR) settings of Application Gateway and the backend server's subnet for any routing anomalies. Make sure the UDR isn't directing the traffic away from the backend subnet. For example, check for routes to network virtual appliances or default routes being advertised to the Application Gateway subnet via Azure ExpressRoute and/or VPN. d. To check the effective routes and rules for a network adapter, you can use the following PowerShell commands: Azure PowerShellCopy

              Get-AzEffectiveNetworkSecurityGroup -NetworkInterfaceName "nic1" -ResourceGroupName "testrg"
           Get-AzEffectiveRouteTable -NetworkInterfaceName "nic1" -ResourceGroupName "testrg"
   

4. If you don't find any issues with NSG or UDR, check your backend server for application-related issues that are preventing clients from establishing a TCP session on the ports configured. A few things to check: a. Open a command prompt (Win+R -> cmd), enter netstat, and select Enter. b. Check whether the server is listening on the configured port. For example: Copy

              Proto Local Address Foreign Address State PID
           TCP 0.0.0.0:80 0.0.0.0:0 LISTENING 4
   

   c. If it's not listening on the configured port, check your web server settings. For example: site bindings in IIS, server block in NGINX and virtual host in Apache. d. Check your OS firewall settings to make sure that incoming traffic to the port is allowed.

Additionally,

Here's what you can try:

1. First validate whether the Application and VM are fine by trying to access the application from another VM in the same subnet.
2. Next, try to get a VM in a different subnet and try to access the application, to mimic the AGW trying to connect to the backend. This will help you validate whether your NSGs are properly configured.

Refer: https://zcusa.951200.xyz/en-us/answers/questions/1851661/resolving-azure-application-gateway-backend-health

If above is unclear and/or you are unsure about something add a comment below.

Please don’t forget to close the thread by clicking "Accept the answer" wherever the information provided helps you, as this can be beneficial to other community members.

Regards,

Ganesh

Hi @Arunkumar Elango

Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.

I understand that you are facing issues with Backend Server Health Status Conflicting with Health Probe Logs in Application Gateway.

• The instance health is displayed as Healthy in the pipeline results. - I don't see any attached image. can you kindly reattach it.

Thanks for confirming the below details -

• A Virtual Machine Scale Set (VMSS) is being used as the backend server.

The health probe is configured correctly to target the expected endpoint.

Network Security Groups (NSGs) and User-Defined Routes (UDRs) appear to be correctly configured, but the error persists.

However kindly confirm the below details -

• Share the output from the connection troubleshoot blade of the application gateway as shown below. Put your URL hostname as it is used on the backend server.
• 
• Kindly ensure you are using the correct backend setting.

1. If you are using different hostname on the backend server as compared to the frontend (listener) ensure you have correct settings configured.

2. Choose no if you backend server is using same hostname as on frontend (Listener) , in short you are not modifying the URL hostname.

• Ensure you have correct custom health probe configured. If you are using HTTPS probe. Make sure you have a complete chain of the certificate configured on the backend server.

Ensure you have a page available under Path on your backend server for example i have given path as /index. If you have a page available on hostname leave path blank.

Choose the correct Backend setting.

Since you have mentioned UDR i assume you are using the supported scenarios as mentioned on the below link. Kindly note

[https://zcusa.951200.xyz/en-us/azure/application-gateway/configuration-infrastructure#supported-user-defined-routes

](https://zcusa.951200.xyz/en-us/azure/application-gateway/configuration-infrastructure#supported-user-defined-routes

)

Kindly note any scenario where 0.0.0.0/0 needs to be redirected through a virtual appliance, a hub/spoke virtual network, or on-premises (forced tunneling) isn't supported for v2.

https://zcusa.951200.xyz/en-us/azure/application-gateway/configuration-infrastructure#v2-unsupported-scenarios

If the below answer addressed your query, please don’t forget to click "Accept the answer" and Up-Vote for the same, which might be beneficial to other community members reading this thread. And, if you have any further query do let us know.

Thanks,