The IPSec process
Applies To: Windows Server 2003, Windows Server 2003 R2, Windows Server 2003 with SP1, Windows Server 2003 with SP2
The IPSec process
This topic provides an overview of IPSec concepts that are central to understanding the IPSec process, including IPSec policy configuration and the Internet Key Exchange (IKE) protocol. In addition, this topic describes how IPSec network traffic processing works, using two intranet computers as an example.
IPSec Policy Configuration
In Windows 2000, Windows XP, and the Windows Server 2003 family, IPSec is implemented primarily as an administrative tool that you can use to enforce security policies on IP network traffic. A security policy is a set of packet filters that define network traffic as it is recognized at the IP layer. A filter action defines the security requirements for the network traffic. A filter action can be configured to: Permit, Block, or Negotiate security (negotiate IPSec).
IPSec filters are inserted into the IP layer of the computer TCP/IP networking protocol stack so that they can examine (filter) all inbound or outbound IP packets. Except for a brief delay required to negotiate a security relationship between two computers, IPSec is transparent to end-user applications and operating system services.
A collective set of IPSec security settings is known as an IPSec policy. Windows 2000, Windows XP, and the Windows Server 2003 family provide a graphical user interface and several command-line tools that you can use to configure an IPSec policy, and then assign it to a computer.
To ensure that IPSec communication is successful and that IPSec meets the security requirements of your organization, you must carefully design, configure, coordinate, and manage IPSec policies. In many organizations, one administrator might be responsible for configuring and managing IPSec policies for many, if not all, computers.
Internet Key Exchange (IKE) security associations
The IKE protocol is designed to securely establish a trust relationship between each computer, to negotiate security options, and dynamically generate shared, secret cryptographic keying material. The agreement of security settings associated with keying material is called a security association, also known as an SA. These keys will provide authenticity, integrity, and optionally, encryption of IP packets that are sent using the security association. IKE negotiates two types of security associations:
A main mode security association (the IKE security association that is used to protect the IKE negotiation itself).
IPSec security associations (the security associations that are used to protect application traffic).
You can configure IPSec policy settings for both types of security associations.
The IPSec service interprets an IPSec policy, expanding it into the components that it needs to control the IKE negotiation. The IPSec policy contains one definition of a packet filter. The packet filter is interpreted in two ways: one uses only the address and identity information to allow IKE to establish a main mode SA (the IKE security association); the other allows IKE to establish the IPSec security associations (also known as quick mode security associations).
IPSec network traffic processing
The following illustration shows how IPSec works in terms of the IPSec components for two intranet computers.
.gif "The IPSec process")
For simplicity, this example is of an intranet in which two computers have an active IPSec policy.
Alice, using a data application on Computer A, sends an application IP packet to Bob on Computer B.
The IPSec driver on Computer A checks its outbound IP filter lists and determines that the packets should be secured.
The action is to negotiate security, so the IPSec driver notifies IKE to begin negotiations.
The IKE service on Computer A completes a policy lookup, using its own IP address as the source and the IP address of Computer B as the destination. The main mode filter match determines the main mode settings that Computer A proposes to Computer B. Computer A sends the first IKE message in main mode, using UDP source port 500, destination port 500. IKE packets receive special processing by the IPSec driver to bypass filters.
Computer B receives an IKE main mode message requesting secure negotiation. It uses the source IP address and the destination IP address of the UDP packet to perform a main mode policy lookup, to determine which security settings to agree to. Computer B has a main mode file that matches, and so replies to begin negotiation of the main mode SA.
Computer A and Computer B now negotiate options, exchange identities, verify trust in those identities (authentication), and generate a shared master key. They have now established an IKE main mode SA. Computer A and Computer B must mutually trust each other.
Computer A then performs an IKE quick mode policy lookup, using the full filter to which the IPSec driver matched the outbound packet. Computer A selects the quick mode security settings and proposes them, and the quick mode filter, to Computer B.
Computer B also performs an IKE quick mode policy lookup, using the filter description offered by Computer A. Computer B selects the security settings required by its policy and compares those settings to those offered by computer A. Computer B accepts one set of options and completes the remainder of the IKE quick mode negotiation to create a pair of IPSec security associations.
One IPSec SA is inbound and one IPSec SA is outbound. The IPSec SAs are identified by a Security Parameter Index (SPI), which is inserted into the IPSec header of each packet sent.
The IPSec driver on Computer A uses the outbound SA to sign and, if required, encrypt the packets. If the network adapter can perform hardware offload of IPSec cryptographic functions, the IPSec driver formats the packets, but does not perform the IPSec cryptographic functions.
The IPSec driver passes the packets to the network adapter driver, indicating whether the adapter must perform the IPSec cryptographic functions. The network adapter transmits the packets into the network.
The network adapter driver at Computer B receives the encrypted packets from the network. The SPI is used by the receiver of an IPSec packet to find the corresponding IPSec security association, with the cryptographic keys required to verify and decrypt the packets. If the network adapter can decrypt the packets in hardware, it verifies whether it can recognize the SPI. If it cannot decrypt the packets in hardware, or if it cannot recognize the SPI, it passes the packets up to the IPSec driver.
The IPSec driver on Computer B uses the inbound SA SPI to retrieve the keys required to validate authentication and integrity and, if required, to decrypt the packets.
The IPSec driver converts the packets from IPSec format back to standard IP packet format. It passes the validated and decrypted IP packets to the TCP/IP driver, which passes them to the receiving application on Computer B.
The IPSec SAs continue to provide very strong, transparent protection for application data traffic. The IPSec SAs are automatically refreshed by an IKE quick mode negotiation for as long as the application sends and receives data. When the application stops sending and receiving data, the IPSec SAs become idle and are deleted.
Typically, the IKE main mode SA is not deleted. By default, the main mode SA has a lifetime of 8 hours. You can configure the main mode SA lifetime to as short as 5 minutes to a maximum of 48 hours. Whenever more traffic is sent, a new quick mode is negotiated automatically to create two new IPSec SAs to protect application traffic. This process is rapid, because the main mode SA already exists. If a main mode SA expires, it is automatically renegotiated as needed.
Any routers or switches in the path between the communicating computers simply forward the encrypted IP packets to their destination. However, if there is a firewall, security router, or proxy server in the path, IPSec and IKE traffic might not be forwarded. These devices must be configured to allow IPSec and IKE protocol packets through. If the IPSec packets are not encrypted, the firewall or security router can still inspect the TCP or UDP ports or other content in the packets. If the contents of these packets are modified after they are sent, the receiving IPSec computer detects the modification and discards the packets.
However, the Windows Server 2003 family implementation of IPSec provides support for a new Internet specification that allows IPSec packets to be modified by a network address translator (NAT). IPSec Encapsulating Security Payload (ESP) packets can pass through NATs that allow UDP traffic. The IKE protocol automatically detects the presence of a NAT and uses User Datagram Protocol-ESP (UDP-ESP) encapsulation to allow IPSec traffic to pass through the NAT.