Front Door responds with OriginTimeout after ~4 seconds despite larger timeout setting

Thanks a ton for the reply. That is unfortunate to hear. I'm going to try going the support ticket route and see if anything comes from it.

We are facing exactly the same issue.

Randomly page elements timeout within about 4secs while the FD timeout is set to the maximum 240 secs. On FD Audit logs it is either 503 or 504 status codes.

Tried removing compressions, adding rule to ignore Accept-Encoding but the issue persists more than a month now.
Opened a ticket and no solution found yet.

PLEASEEEEEE ANYONE NEED ASSISTANCE ON THIS !!!

Pulled down the log from today and there was a single instance of timeout with the same symptoms - FD reports timeout, 504, and of course the request never made it to the origin. So this is still happening, though less frequently. At least it failed faster than 4 seconds :D

"timeToFirstByte": "2.832",

"timeTaken": "2.832",

"httpStatusCode": "504",

"httpStatusDetails": "504",

"pop": "MNZ",

"cacheStatus": "N/A",

"errorInfo": "OriginTimeout",

"ErrorInfo": "OriginTimeout",

The issue is for items that are in the cache as it seems.

I tested the same endpoint on 2 domains, one with caching on FD and the other without. On the one that I had the caching enabled I had the 504 timeouts in the audit log while on the other domain did not.

Also noticed that for every time 504 happens in the log there are 2 entries at the same time. One with sni_s my hostname and another with sni_s == "originshield|parentcache|https|tier2"

Still have ticket with Microsoft and hope to have a solution on this. Will keep you posted.

New update. Since last night I have tried different settings on FD. The issue happens only when caching is enabled. Disabled caching and purged cash on the domain and stopped happening. Will continue with the ticket with MS since caching is needed for sure. Once I have any new updates I will comment.

New Update.

My Setup tested:

Azure Frontdoor --> Fortigate (Policies with Virtual IPs) --> VM on Azure (IIS)

Setup 1) FD Caching enabled --- ERRORS 504

Setup 2) FD Caching disabled ---- NO ERRORS 504

New Setup Tested bypassing Fortigate Firewall:

Azure Frontdoor --> VM on Azure (IIS)

Setup 1) FD Caching enabled --- NO ERRORS 504

Setup 2) FD Caching disabled ---- NO ERRORS 504

Interesting! In our case, we didn't have a firewall, it was FD -> LB pool -> VM

We have had the same 4 second timeout problem with Front Door. Caching is disabled. No firewall inbetween FD and origins.

The problem occurs across multiple origin types: App Service, Linux VMSS + public LB, Windows VMSS + private link LB, Cloud Services (Extended Support).

We have a pending ticket with support since June 10.

• @GitaraniSharma-MSFT I'm working with the customer who made the last response to this thread. Is this something we are seeing with more customers broadly? Thanks in advance for your time.

Hello @Joshua Waddell , yes, we are seeing this issue with more customers.

It looks like an upgrade in few of the backend environments has caused this issue and a support request is required to fix it from the backend.

Regards,

Gita

@GitaraniSharma-MSFT If we reach out to the support about this issue, is there an existing KB #/ticket #/service bulletin to refer to, so we don't have to reexplain the entire thing again?

@GitaraniSharma-MSFT We are having the same issue too where the request is timing out at 4 seconds. Is there a solution to this?

Hello @Sergey Stoma , there is an internal incident (not shareable publicly).

@Asjana , Yusef , for the recent issue/bug which started in April last week, the Product Group team shared the below RCA:

Incident Summary  

Starting April 24th at 8:50 UTC, several customers started reporting a dip in their backend health status. The problem continued till May 10th as we investigated the root cause. 

Upon investigation it was observed that health probes from AFD to backends were facing intermittent connection failures, which started showing up as a dip in the backend health metric. Even though there were health probe failures, there was no impact on the regular traffic of customers.

Root Cause 

Starting from the last week of April, a scheduled OS upgrade was being rolled out to several AFD environments. Due to a bug in the newer OS version, a system configuration parameter was not set to an appropriate high value.

This parameter is usually set to an appropriate high value in anticipation of the higher load the health probe service (in available machines) will face when a subset of AFD machines is temporarily taken offline during OS upgrades.  

But because of the new default lower value set by the bug, the health probe service started hitting the limits when machines were being taken offline for the upgrade. This caused intermittent connection failures in the health probes among a subset of customer origins. 

Mitigation  

To mitigate the problem, we overrode the lower default value introduced by the bug and set the appropriate high limit needed for the health probe service. We will roll out a permanent fix for the bug by the end of June. 

Next steps and repair items

We deeply apologize for this incident and for any inconvenience it has caused. In our continuous efforts to improve platform reliability, we will be performing below repairs   

• Identify system level settings that had lower default values in the new OS version and configured more appropriate values for them.
• Add metrics to track connections usage in health probe service and monitoring based on it.
• Roll out the fix for the bug in the latest OS version. [rollout completion end of June]
• Add monitoring to detect changes in configured limits in subsequent OS updates [end of June] 

The above fix was supposed to be rolled out by end of June, but I will check with the Product Group team for any changes in the ETA.

Regards,

Gita

HI @GitaraniSharma-MSFT, thanks for the update. In our case the Health probes are disable for this backend pool and there is only one backend setup for it.

Hi @GitaraniSharma-MSFT we are still experiencing this 4 second timeout issue across multiple origins and front door instances. This problem occurs on origin groups where Health Probes are disabled.

On July 3rd over 24 hours, 0.09% of requests to front door failed with a 504 response at 4 seconds.

On July 2nd over 24 hours, 0.1% of requests to front door failed with a 504 response at 4 seconds.

We obtain these percentages by querying the Front Door logs and comparing the total # requests to the # requests where ('properties.httpStatusCode'=="504" AND tonumber('properties.timeToFirstByte') < 4.1)) (this is splunk query syntax).

Is it possible for the Front Door Product Group analyze for this behavior across customers to determine how wide spread the problem is? And to understand whether changes cause issues?

Also from the RCA:

 Even though there were health probe failures, there was no impact on the regular traffic of customers.

This is not correct. Our customers are seeing 504 gateway timeout issues in their web browsers and our Pingdom checks are going down frequently.

Hello @Asjana , Yusef and @J K ,

The issue that began in the last week of April (for which I provided an RCA above) has been confirmed fixed by the Product Group team. All Azure Front Door edge servers and environments have been updated and fix rollout is complete.

As per the Azure Front Door health probe document,

If you have a single origin in your origin group, you can choose to disable health probes to reduce the load on your application. NOTE: If there is only a single origin in your origin group, the single origin will get very few health probes. This may lead to a dip in origin health metrics, but your traffic will not be impacted.

However, if you are experiencing an actual impact on your traffic, further investigation is required, including the collection of backend logs. Therefore, I recommend opening a support request.

Regards,

Gita

Hi @GitaraniSharma-MSFT, thanks for the update. We have a support ticket opened with MS and just like @J K mentioned we as well have our Health Probes disabled for this origin but we continue to see the issue. This doesn't seem like an isolated issue given that there are other customers seeing this behavior. Could you please escalate this issue to the Product Group team.Thanks

Hi @GitaraniSharma-MSFT appreciate the update.

Similar to @Yusef Asjana the issue is still ongoing for my company.

We have a support ticket open since June 10. Front Door Product Group said that this fix would resolve the issue but the issue is still ongoing.

Unfortunately we aren't getting any traction in our support request to investigate the root cause of the issue. I don't understand how a proxy can return a 504 timeout error after 4 seconds? Any clues as to the root cause would be helpful, if there's something we can try on our end?

We do not have health probes enabled on origins. Example configuration for an origin:

This issue is occurring for multiple origins, examples:

• Apache on Azure VMSS (Linux)
• Varnish container on Azure Kubernetes Service
• IIS on Azure VMSS (Windows)

This issue is occurring across multiple POPs, example of the 4 second timeout requests logged in the past 3 hours:

Top 10 Values	Count	%	 
DFW	4	18.182%	
AKL	3	13.636%	
YVR	3	13.636%	
AM	2	9.091%	
FRA	2	9.091%	
SIN	2	9.091%	
ATL	1	4.545%	
BN	1	4.545%	
CH	1	4.545%	
DM	1	4.545%	

Hello @Yusef Asjana and @J K ,

I've reached out to the Azure Front Door Product Group team for any latest updates on this issue. Will keep you posted.

In the meantime, could you please let me know which SKU of Azure Front Door you are using? Is it Classic or Standard/Premium?

Also, @J K , could you please share the support request number, if possible?

Regards,

Gita

Hi @GitaraniSharma-MSFT , we are using the Classic Front Door.

Hi @GitaraniSharma-MSFT

We are using Azure Front Door Premium.

Our support request number is TrackingID#2406080030000792

Hello @Yusef Asjana and @J K ,

Thank you for the details.

I've reached out to the Azure Front Door Product Group team regarding this issue and awaiting their updates. Will keep you posted.

Meanwhile, could you please check the KeepAliveTimeout configuration in your origins and Origin response timeout configuration in your Azure Front Door (The ability to configure Origin response timeout is only available in Azure Front Door Standard/Premium).

Refer: https://zcusa.951200.xyz/en-us/answers/questions/1329800/intermittent-503s-on-azure-front-door-from-origini

https://zcusa.951200.xyz/en-us/azure/frontdoor/troubleshoot-issues#503-or-504-response-from-azure-front-door-after-a-few-seconds

Also, check if the client is sending byte range requests with Accept-Encoding headers.

Regards,

Gita

We have seen the same issue since this morning - timeTaken 4sec and timeToFirstByte 4sec

Hello @GitaraniSharma-MSFT

We are also experiencing this issue for roughly 1% of customer traffic.

The exact same issues that are being mentioned in this this thread with OriginTimeout error codes, alongside roughly 4 second timeouts.

We've changed the origin timeout within Front Door to 230 seconds, which hasn't seen any improvements. We've also applied the fix to remove the Accept-Encoding header that you suggested and nothing has changed.

Is there any further developments since the last update for what is causing this, and also any potential fixes?

I can confirm we are also seeing this issue.
Seems to be affecting roughly 1% of traffic for our customers, which is extremely problematic.

@GitaraniSharma-MSFT is there any update on this?

I am getting timeout from azure classic frontdoor in 4-5 secs.
Is fix deployed for this issue?