Share via

Configure Settings to Optimize IPsec Behavior on Earlier Versions of Windows

  • Article

Applies To: Windows 7, Windows Essential Business Server, Windows SBS 2003, Windows SBS 2008, Windows Server 2000, Windows Server 2003, Windows Server 2003 R2, Windows Server 2003 with SP1, Windows Server 2003 with SP2, Windows Server 2008, Windows Server 2008 R2, Windows Vista

We recommend that you use Group Policy to include the following registry settings before deploying IPsec rules to computers running Windows Server 2003 or Windows XP.

The first setting changes the protocols that are exempted from IPsec by default. This setting is documented in article 810207 in the Microsoft Knowledge Base (https://go.microsoft.com/fwlink/?linkid=110516). We recommend setting the value to 2 to exempt RSVP, Kerberos, and ISAKMP protocol traffic, but not multicast or broadcast traffic.

The second setting is for configuring simplified IPsec policy. This setting is documented in article 914841 in the Microsoft Knowledge Base (https://go.microsoft.com/fwlink/?linkid=110514). We recommend that you set the bits ‘0x14’ in the IKEFlags setting. This value is interpreted as a bitmap, so if the existing value is not 0, then you should take steps to preserve the existing bits and add these bits to the existing value. If the value is initially a 0, then you can simply write the value 0x14 to the registry key. If you are running Windows Server 2003 with Service Pack 2 (SP2) or earlier or Windows XP with SP2 or earlier, then you must first install the update documented in article 914841 in the Microsoft Knowledge Base to enable simplified IPsec policy (https://go.microsoft.com/fwlink/?linkid=110514). The update is included with Windows XP with SP3.

The next setting is to support IPsec over NAT-T when the client or the server is behind a network address translator. This setting is documented in article 885407 in the Microsoft Knowledge Base (https://go.microsoft.com/fwlink/?linkid=119888). By default, IPsec connections are blocked if a computer is separated from the other computer by a NAT device. If one of the computers must be behind a NAT device, then use one of the values supported by the documented registry key to enable the appropriate scenario. If you do not have NAT devices separating computers that must communicate by using IPsec, then set the value to 0.

The final setting is for performance; it configures the IP stack to dynamically discover the largest packet size available between two communicating computers, instead of using a potentially inefficient default, fixed size. This setting is already set to the correct value in Windows Vista, but for earlier versions of Windows, and to ensure that it cannot be changed on the client computers, set the value to 1. The setting is documented in EnablePMTUDiscovery (https://go.microsoft.com/fwlink/?linkid=119891).

To support the changing of registry settings by using a GPO, you must first configure the Group Policy Management Console (GPMC) with a custom .xml template file that defines the registry settings to the Group Policy Management Editor. You can use the sample file provided in Appendix A: Sample GPO Template Files for Settings Used in this Guide in the Windows Firewall with Advanced Security Design Guide. The sample file also includes recommended values for two other settings that are useful for configuring IPsec on the network.

Administrative credentials

To complete these procedures, you must be a member of the Domain Administrators group, or otherwise be delegated permissions to modify the GPOs.

To configure the custom .xml file to use the settings in GPMC

  1. Open the folder containing the custom .xml file. If the file does not yet exist, you can use the sample file shown in Appendix A: Sample GPO Template Files for Settings Used in this Guide in the Windows Firewall with Advanced Security Design Guide as a starting point. Copy the source data into Notepad, and then save the file with an .xml extension to your desktop. Before you use the file, make sure that you customize it to meet your organization’s requirements.

  2. Click Start, click Administrative Tools, and then click Group Policy Management.

  3. Find the GPO to which you want to add the custom registry settings, right-click it, and then click Edit.

  4. In the Group Policy Management Editor, expand Computer Configuration, expand Preferences, and then expand Windows Settings.

  5. Drag and drop your custom .xml file onto the Registry node in the navigation pane.

  6. When asked to confirm that you want to import the document, click Yes.

  7. Click the Server and Domain Isolation Settings folder to see the individual settings. The sample file uses the recommended values for each registry key as the default value. Change the registry keys as required for your environment before deploying the GPO. To change one, double-click the setting, and on the General tab, change Value data.

  8. Link the GPO to the appropriate container or containers and use WMI filters to restrict application of the GPO to only those computers that require the setting.

    • For an isolated domain, you might consider assigning the GPO to the domain container and using a WMI filter to apply it to only computers running Windows XP or Windows Server 2003.

    • For an isolated server environment without an isolated domain, you can assign the GPO to only those computers that must access the isolated server. To do this, assign the GPO to the domain container, use WMI filters to restrict the GPO to only those computers running Windows XP or Windows Server 2003, and then place the group that grants access to the isolated server in the security group filter for the GPO.

If you arrived at this page by clicking a link in a checklist, use your browser’s Back button to return to the checklist.