Network latency between Azure Global VNet Peering

Hello @SHAKIR SHAIKH

Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.

I understand that you are facing issues with intermittent drops while accessing your website site1.abc.com and traffic path is given as below.

East US users ---> VPN/AVD ---> Az. Workload VNet----> Route to Azure Firewall VNet----> Global VNet peering between Azure Firewall VNet and Central US VNet---> Citrix Netscaler (Webpage)

Since the issue is intermittent it is advised to perform the troubleshooting during the time when you experience slowness on the site.

• Also, it is also important to fist identify at which layer issue is happening as you correctly mentioned in your question. Hence i would request you perform the below Linux commands from a machine in East US site.

end=$((SECONDS+300)); while [ $SECONDS -lt $end ]; do curl -w "dns_resolution: %{time_namelookup}, tcp_established: %{time_connect}, ssl_handshake_done: %{time_appconnect}, TTFB: %{time_starttransfer}\n" -o /dev/null -s https://site1.abc.com; sleep 1; done

This command will run in loop for 5 minutes and will execute after every 1 second sleep timer you may adjust the timers.

• These commands will share you the time taken to dns resolution, 3 way tcp established time (Network layer) , SSL Handshake time ( TLS layer) and TTFB( Application layer ).
• During slowness these timers will assist us to identify the Layer where we need to focus for our further troubleshooting.
• During the issue time we can test the Global Vnet peering connectivity using the Network Watcher kindly refer the below page for help -
• https://zcusa.951200.xyz/en-us/azure/network-watcher/connection-troubleshoot-portal
• It is also advised to run the Packet captures on the Source client VM , Azure Firewall and Server where site is hosted simultaneously.
• If you have Wireshark preinstalled, you may use the same or you may refer the below page to capture without installing the Wireshark.
• https://techcommunity.microsoft.com/t5/iis-support-blog/capture-a-network-trace-without-installing-anything-amp-capture/ba-p/376503#:~:text=1.%20Open%20an%20elevated%20command%20prompt%20and%20run%3A,a%20temp%20directory%20or%20choose%20another%20location%29.%202.

Kindly update the outcome once you have required logs so that we can analyze further to understand the issue.

@SHAKIR SHAIKH ,

Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.

May I ask how you are isolating this to be a network issue?

• Both in terms of connectivity and latency
• It is possible that the application/server is over utilized or the issue is application related.

With that said,

1 . Instead of testing this with remote devices/users, I would suggest you deploy a dummyVM in the AzureEastVnet and try to access the site via VNET Peering.

• If the issue prevails, we can eliminate that the the issue is related to S2S or AVD.
• You can run a TCPPing or traceroute during the time of the issue and see where the latency is.

2 . If the issue is not happening in this dummyVM,

• Is this a new set up? (AzureEast users)
• If this was an existing set up, did you make any recent changes that resulted in the latency/connectivity issue?
• Do you observer a pattern/time frame during which the issue is prevalent?
• Are you seeing any application logs for the same time frame.
• What is the latency you are experiencing when the issue occurs vs when the issue does not occur?

NOTE :

• If you'd like to know the expected latency between regions over Microsoft backbone network (not VPN over Internet), you can refer to Azure network round-trip latency statistics

Cheers,

Kapil

@UJTyagi-MSFT We do not have any Linux VMs accessing the site. Only users who connect through VPN or AVD are windows hosts.
If you can share the command that can run on windows client machines, then I can provide the logs.

Hi KapilAnanth-MSFT,

Thanks for the getting back. I'm still unsure if this is a network latency/application related issue and hence, want to isolate it. Regarding answers to your questions PFB:

• Both in terms of connectivity and latency
  • No issue with connectivity is observed. After accessing the application for like 40mins - 1 Hr all the application users from the EU site experience session freezing issue and eventually the session disconnects and resumes in couple of mins.
• It is possible that the application/server is over utilized or the issue is application related.
  • Citrix netscaler which is acting as LB, has 2 backend hosts in active-passive mode and the Central US team confirms that utilization isn't even reaching 50% on the host.

1 . Instead of testing this with remote devices/users, I would suggest you deploy a dummyVM in the AzureEastVnet and try to access the site via VNET Peering.

• If the issue prevails, we can eliminate that the the issue is related to S2S or AVD.
  • I'm going to deploy another testing VNET and test the connectivity and will share the results with you.
• You can run a TCPPing or traceroute during the time of the issue and see where the latency is.
  • TCPPing and traceroute isn't helping much as the latency/connectivity drop is experienced only when accessing the application.
• Is this a new set up? (AzureEast users)
  • The setup has been in place for a couple of months but issue started to arise when the number of users who are accessing the application started to increase.
• If this was an existing set up, did you make any recent changes that resulted in the latency/connectivity issue?
  • No changes were made, only new users were onboarded to access the application.
• Do you observer a pattern/time frame during which the issue is prevalent?
  • During peak hours, when most of the users are accessing the application. No issues observed when no one is accessing the application.
• Are you seeing any application logs for the same time frame.
  • Working to see if application logs can be accessible from citrix.
• What is the latency you are experiencing when the issue occurs vs when the issue does not occur?
  • While accessing application - 4-7 secs latency is observed.
  • When issue is not observed - 20-30 ms.