Share via

Azure Application Gateways do not resolve Private Endpoints of storage accounts via custom DNS servers

Cat Mucius 91 Reputation points
2022-01-29T11:32:43.21+00:00

Good day,

I got a couple of setups looking like this:

  • An Application Gateway ( WAF V2 or Standard V2) is placed in a VNET configured with custom DNS servers (They were restarted after the last change in this DNS servers list, so they are updated).
  • These custom DNS servers are used to resolve addresses of Private Endpoints - for App Services and for static websites on Storage Accounts - and they resolve them successfully:
    appservicename.azurewebsites.net ==> appservicename.privatelink.azurewebsites.net ==> private IP
    staticsitename.z6.web.core.windows.net ==> staticsitename.privatelink.web.core.windows.net ==> private IP
  • Private Endpoints of both these types are defined as backend pools of Application Gateways.
  • The Application Gateways work perfectly well with Private Endpoints of App Services. If I capture traffic on the DNS servers, I can see queries for these App Services names sent after the App Gateways are started. These names are resolved, health probes succeed and traffic is forwarded to these App Services. All's fine.
  • However, the App Gateways don't even try to resolve names of Private Endpoints of Storage Accounts (for blob service or for static website service). I don't see any queries about their names arriving to our custom DNS servers. These names are resolved to their public IPs, and App Gateways reach these backends via another route - by Service Endpoints. So the App Gateways' subnet must be listed as allowed at the Firewalls and virtual networks page of these Storage Accounts, otherwise the health probes fail with error 404 (Private Endpoints shouldn't be affected by these settings).
  • I made an experiment:
    a.) defined a Private DNS Zone for these Private Endpoints privatelink.web.core.windows.net and added to it relevant records resolving to private IPs;
    b.) bound it to the VNET where an Application Gateway resides;
    c.) stopped & started the Application Gateway;

Only after that the App Gateway resolved the staticsitename.z6.web.core.windows.net record to its Private Endpoint IP and was able to reach the Storage Account regardless of its Firewalls and virtual networks page settings.

So as I understand:

  • Application Gateway uses custom DNS servers assigned to its VNET to resolve most of its backend servers FQDNs.
  • However, for some of them (like Storage Accounts names under the windows.net domain) it uses the Azure built-in DNS service (168.63.129.16), ignoring custom DNS.
  • So the only way to make an App Gateway resolve correctly addresses of Private Endpoints for Storage Accounts (and possibly other PaaS resources) is to attach a Private DNS Zone to its VNET, making the built-in Azure DNS service to read records from this Zone.

I didn't find anything like this in documentation.

Questions:

  • Is this correct?
  • If it is, what's the logic behind this way of splitting DNS queries? Is there any way to disable it?

Thanks,
Mucius.

Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
3,330 questions
Accepted answer
  1. SaiKishor-MSFT 17,321 Reputation points
    2022-02-18T19:20:06.683+00:00

    @Cat Mucius

    Your observations are correct; we pin a subset of FQDNs to always utilize Azure DNS (168.63.129.16 address). This is as per design due to Azure's internal architecture. As a workaround, customers can associate the privatelink DNS zone to the virtual network, regardless if they are utilizing custom DNS servers on the Vnet or not. This experience will be improved in an upcoming feature, targeted for the back half of this year. Hope this helps.

    Please let us know if you have any further questions and we will be glad to assist you further. Thank you!

    Remember:

    Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.

    Want a reminder to come back and check responses? Here is how to subscribe to a notification.

    1 person found this answer helpful.

2 additional answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Vidya Narasimhan 2,126 Reputation points Microsoft Employee
    2022-01-30T15:16:46.74+00:00

    @Cat Mucius Whether it is App Service or Storage, private endpoint name resolution for Private Link resources can be done via Azure Private DNS or your DNS forwarder that forwards DNS queries to the recursive resolvers in Azure provided via the virtual IP 168.63.129.16 .
    Please go through some of the best practices provided in the below link
    https://zcusa.951200.xyz/en-us/azure/cloud-adoption-framework/ready/azure-best-practices/private-link-and-dns-integration-at-scale#private-link-and-dns-integration-in-hub-and-spoke-network-architectures
    You can create private dns zones for the private link resources. The DNS Server Vnet can be linked to the private DNS Zones. The DNS servers use the Azure-provided DNS resolver (168.63.129.16) as a forwarder. All workload VNets can use the DNS servers hosted as the primary and secondary DNS servers.

    0 comments
  2. Jeroen Bonsel 6 Reputation points
    2022-11-07T10:02:19.28+00:00

    Also do not forget to enable NETWORK POLICY FOR PRIVATE ENDPOINTS.
    Even if you have dns resolution working, Without the network policy, route tables to private endpoints are ignored.

    0 comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.