Azure Firewall has started NATing random traffic flows between VMs

Duncan Sinclair 0

Our monitoring system found a web site not responding last Saturday morning (24th August). Logs showed that it could no longer talk to its database.

The web site is running on a VM -- VM1. The SQL Server is on VM2.

They are on different subnets of the same Vnet, but routed through a 'premium' Azure firewall.

Rules on the firewall, and subnet NSGs allowed traffic, and the activity logs showed there have been no changes to this.

Firewall logs showed traffic was being allowed.
Examining NSG flow logs for the two subnets showed that traffic was leaving VM1's subnet successfully (we also saw this on the firewall logs) but the flow log on VM2's subnet did not show this traffic - at least not exactly.

Instead it showed SQL traffic from the firewall's IP addresses being blocked by the NSG as we wouldn't expect SQL traffic to come from the firewall.
It appears that the firewall is incorrectly NAT'ing this traffic flow. (But not non-SQL traffic between the same two hosts.)

Allowing SQL traffic from the IP addresses of the firewall allowed connectivity to be restored between the two VMs.

Since then we have come across a second flow between two different VMs and on port 443 that also seems to be incorrectly NAT'ed by the firewall.

Anyone got any suggestions as to how we could fix this?

KapilAnanth-MSFT 46,096 Reputation points Microsoft Employee

2024-09-13T04:31:01.26+00:00

@Duncan Sinclair ,

This is a strange behavior - traffic for a specific port arriving with Firewall's IP while rest of the ports are not NATed.

To troubleshoot further, we will need a specialized 1:1 session, where a support engineer can and check the backend logs to pinpoint the issue. If you have a support plan you may file a support ticket, else please do let us know, we will try and help you get a one-time free technical support.

P.S : We will be able to provide you one-time support if and only if your subscription is not managed by a CSP

Cheers,

Kapil
Duncan Sinclair 0 Reputation points

2024-09-13T08:38:17.2866667+00:00

We use a CSP, so I shall raise a support ticket with them. I posted here to see if there was anything obvious I could do to fix the issue, and if there were anybody else seeing the same issue.

Many thanks Kapil.
KapilAnanth-MSFT 46,096 Reputation points Microsoft Employee

2024-09-16T04:28:45.2866667+00:00

@Duncan Sinclair , sure.

Let us know how this goes

Cheers,

Kapil

1 answer

KapilAnanth-MSFT 46,096 Reputation points Microsoft Employee

2024-09-03T11:34:28.9966667+00:00
@Duncan Sinclair ,

Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.

Can you confirm if the SNAT is "Random" or "Happening always suddenly" ?

Is this happening only with the above environment

WebVM to DB VM

or there are other traffic exhibiting this behavior as well

It is possible that SNAT private IP address ranges is configured to SNAT VNet traffic as well (assuming the VNET is using RFC 1918 ranges)

Can you share a screenshot of what this setting is?

Can you confirm Auto-learn SNAT routes preview is not enabled ?

Cheers,

Kapil
Please sign in to rate this answer.
Duncan Sinclair 0 Reputation points

2024-09-11T13:21:09.04+00:00

Hi Kapil, thanks for your reply.

By "Random", I mean it has started happening with two different traffic flows unexpectantly.

So far, this has only been between VMs in different subnets of the same vnet.
SNAT private ranges is configured to exclude the usual RFC 1918 ranges, plus our on-site range. All our Azure infrastructure has IPs within the 10.0.0.0/8 range.

Auto-learning SNAT routes preview in not enabled.

KapilAnanth-MSFT 46,096 Reputation points Microsoft Employee

2024-09-12T04:28:37.71+00:00

@Duncan Sinclair ,

This looks like an isolated issue or related to the implementation of the connecting protocol.

From this WebVM to DB VM,

Can you do a TCP Ping or PSPing and check the Firewall logs?

Can you collect packet capture on the source and destination and confirm the destination received traffic from Azure Firewall's Private IP?

Cheers,

Kapil

Duncan Sinclair 0 Reputation points

2024-09-12T08:52:06.8533333+00:00

Using 'Test-NetConnection' from the web VM failed to connect.

Logs from the firewall showed the connections as "Allow". No suggestion of NAT happening.

Logs from the NSG of the web VM's subnet showed the connection attempts leaving the VM, heading out of the network to ports 445 and 1433 on the DB VM.

Logs from the NSG of the DB's subnet showed traffic arriving from the web VM to port 445 (allowed), and traffic to port 1433 coming from one of the firewall's private IPs (denied).

At this point I added a new rule to the NSG on the DB's subnet to allow this traffic coming from the firewall's address and everything started working.

KapilAnanth-MSFT 46,096 Reputation points Microsoft Employee

2024-09-13T04:29:03.9766667+00:00

@Duncan Sinclair ,

This behavior is strange - a specific port using Azure Firewall's IP while others are not NATed.

To troubleshoot further, we will need a specialized 1:1 session, where a support engineer can check the backend logs to pinpoint the issue.

If you have a support plan you may file a support ticket, else please do let us know, we will try and help you get a one-time free technical support.

P.S : We will be able to provide you one-time support if and only if your subscription is not managed by a CSP

Cheers,

Kapil
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Azure Firewall has started NATing random traffic flows between VMs

1 answer

Your answer