Share via

Step 1: Creating a Connection Security Rule that Requests Authentication

  • Article

Updated: December 7, 2009

Applies To: Windows 7, Windows Server 2008, Windows Server 2008 R2, Windows Vista

In this step, you create connection security rules for the contoso.com domain that cause all member computers to require authentication for inbound network traffic, and request authentication for outbound traffic. You start by using a GPO that only requests inbound authentication, and after you confirm that it is working you revise it to require inbound authentication.

Specifying the IPsec algorithms to use

For simplicity, the rules that you create in the following procedures use the default IPsec main mode and quick mode settings, which specify certain combinations of integrity and encryption algorithms to be included in negotiations. However, Windows provides lots of flexibility in configuring the specific main mode and quick mode algorithms that are used for any given connection. All of the computers that you want to be able to communicate with each other must support at least one set of algorithms in common. If you must use a specific combination of algorithms then you can do one of the following:

  • Alter the global IPsec default values. In your GPO, open the Windows Firewall with Advanced Security Properties page, and in the IPsec defaults section, click Customize. You can configure the algorithms used to negotiate protection for both the main mode and quick mode security associations (SAs), and the authentication options available. Changing these settings alters them for all IPsec connections made to and from this computer whose connection security rules do not specify otherwise, and that do not match a main mode rule.

  • Create a connection security rule with specific quick mode settings. By using the netsh advfirewall consec add rule command, you can create a connection security rule that includes specific quick mode algorithm combinations. If you specify these in the rule then they are used instead of those in the global IPsec default settings. Use the qmsecmethods parameter. One quick mode option added to Windows 7 and Windows Server 2008 R2 is “Null encapsulation”. This option specifies that no integrity protection is provided to each network packet in the connection. No AH or ESP header is used to encapsulate the data. This option is provided for compatibility with network equipment and software that is incompatible with AH or ESP. You can specify the use of Null encapsulation in either the global IPsec defaults (not recommended), or in a connection security rule that matches only the network traffic that must use Null encapsulation. For more information about how to create connection security rules with custom quick mode settings, see Netsh AdvFirewall Consec Commands (https://go.microsoft.com/fwlink/?linkid=157374).

Note

You cannot create a rule with specific quick mode settings by using the Windows Firewall with Advanced Security MMC snap-in.

  • Create a main mode rule. Starting with Windows 7 and Windows Server 2008 R2, you can create a rule that specifies main mode encryption, integrity, and authentication settings. Connections that match a main mode rule use the main mode rule settings instead of the settings specified in the connection security rule or the global IPsec defaults. To create a main mode rule, use the netsh advfirewall mainmode add rule command. For more information, see Netsh AdvFirewall MainMode Commands (https://go.microsoft.com/fwlink/?linkid=147508).

Firewall and Connection Security integration

One of the big advantages of integrating the firewall functionality with IPsec is the additional options available on a firewall rule. In Windows Vista and later versions of Windows, you can now create firewall rules that filter network traffic based on the following criteria:

  • Allow the connection if it is authenticated and integrity-protected. Network traffic that matches this firewall rule must be protected by an IPsec connection security rule that requires both authentication of the connection and the use of an AH or ESP algorithm that helps protect the integrity of each network packet in the connection.

  • Require the connection to be encrypted. Network traffic that matches this rule must be encrypted by using an IPsec connection security rule that specifies using ESP encapsulation with encryption. The network traffic must also be authenticated and integrity-protected.

    The additional option to Allow the computers to dynamically negotiate encryption enables you to deploy fewer connection security rules to your client computers. For example, to enable encryption for a single port on a server while network traffic to all other ports on the server are unencrypted previously required two rules on both the server and all its clients: one generic rule that applied to all traffic to a server and did not require encryption, and one that specified the server’s IP address and the required port number on the server, and that required encryption. As the number of ports and the number of servers that require this special handling grow, the task of creating and maintaining the required connection security rules becomes more difficult.

    By using this option, only the generic rule that requires authentication and integrity is required on the clients. On the server only, you create a firewall rule for the specific port number that requires encryption, and that enables dynamic encryption. The main mode SA can therefore be negotiated by using the single rule on the client. When the client sends traffic to the specified port number protected by the firewall rule, the server starts a quick mode negotiation with the client to create an “upgraded” SA that requires encryption. Any encryption algorithm combinations enabled in the global IPsec defaults for quick mode negotiations can be used. In addition to being able to specify a port, you can alternatively create a firewall rule that requires encryption only for a specified service, program executable, or protocol. All network traffic to other services, other programs, or that use other protocols on the computer would not trigger a quick mode SA negotiation to require encryption.

    Although this option was introduced with Windows 7 and Windows Server 2008 R2, it is compatible with computers that are running Windows Vista and Windows Server 2008 and can be applied to them by using Group Policy.

  • Allow the connection to use null encapsulation.. This option specifies that after main mode negotiation and authentication is completed, the quick mode SA requires no AH or ESP encapsulation. The data stream for the connection therefore does not receive per-packet integrity protection. This option is provided for compatibility with network equipment or software that is incompatible with AH or ESP.

To create a new GPO for domain isolation

  1. On MBRSV1, in Group Policy Management, right-click Group Policy Objects, and then click New.

  2. In Name, type Domain Isolation, and then click OK.

  3. In the navigation pane, right-click your new GPO, and then click Edit.

  4. In Group Policy Management Editor, in the navigation pane, right-click the top node for your Domain Isolation GPO, and then click Properties.

  5. Select the Disable User Configuration settings check box, because this is a computer-only GPO.

  6. In the Confirm Disable dialog box, click Yes, and then click OK.

  7. In the navigation pane, expand Computer Configuration, expand Policies, expand Windows Settings, expand Security Settings, expand Windows Firewall with Advanced Security, and then expand Windows Firewall with Advanced Security - LDAP://cn={GUID},cn=policies,cn=system,DC=contoso,DC=com.

  8. Right-click Connection Security Rules, and then click New rule.

  9. On the Rule Type page, click Isolation, and then click Next.

  10. On the Requirements page, confirm that Request authentication for inbound and outbound connections selected, and then click Next.

Warning

In a production environment, we recommend that you set request mode first and allow the GPO to fully propagate to the network. Confirm that all your computers are communicating successfully by using IPsec before changing the rules to require mode. Setting the rule to require mode first can result in computers that cannot communicate until all the computers receive and apply the GPO. In a later step, you modify the rule to change it to require inbound authentication.

  1. On the Authentication Method page, click Computer and user (Kerberos V5), and then click Next.

Note

Although you typically do not filter by user credentials in domain isolation, user authentication in addition to computer authentication enhances security, and makes implementing server isolation based on a user’s identity later much easier, because a compatible authentication method is already in place.

  1. On the Profile page, clear the Private and Public check boxes, and then click Next.

  2. On the Name page, type Request Inbound Request Outbound, and then click Finish.

Next topic: Step 2: Deploying and Testing Your Connection Security Rules