Written by

greg jorgensen

Date published

May 27, 2023

• What is an IPsec VPN?
• IKE
• Transport or Tunnnel Mode
• IKE (Internet Key Exchange) v1
• IKEv2
• Authentication Header (AH) & Encapsulating Security Payload (ESP)
• Establishing an IPsec VPN
• Phase 1
• Phase 2
• Additional details:
• Cryptographic Algorithms
• Troubleshooting?

What is an IPsec VPN?

A Virtual Private Network, or VPN, is named as such because it creates a secure, private network connection over a public network infrastructure, typically the Internet.

IPsec is a “suite” of protocols - a framework of open standards that can work together to provide a CIA and non-repudiation for IP packets. IPsec is not an actual protocol itself.

These protocols are:

• Internet Key Exchange (IKE): Negotiates and provides cryptographic keys.

ISAKMP defines the procedures for authenticating a communicating peer, creation and management of Security Associations, key generation techniques, and threat mitigation.

Oakley: This key-agreement protocol allows the authenticated parties to exchange keying material across an insecure connection using the Diffie-Hellman algorithm.

SKEME: A versatile key exchange technique which provides anonymity, repudiability, and quick key refreshment.

• Authentication Header (AH): This provides packet-level authentication and optionally provides anti-replay services. It essentially allows the receiver to verify that the data was sent by a claimed sender (authentication).
• Encapsulating Security Payload (ESP): This provides confidentiality, along with optional data-origin authentication, connectionless integrity, and anti-replay service. Essentially, it provides encryption for the payload.

(IKE is generally used all the time but is not required, keys can be manually configured)

IKE = Key exchange

AH = Authentication

ESP = Encryption

While IPSec is commonly used in VPNs to create a secure tunnel between two networks over the internet, it's not limited to this use case.

IPSec operates at the Network (or internet) layer of the TCP/IP

This means it can provide security for any type of protocol riding on top of IP, including TCP, UDP, ICMP, and the data payloads they carry.

IPSec can secure HTTP, FTP, SMTP, Telnet, and other application layer protocols, regardless of whether they already have built-in security measures like HTTPS.

👉

Network-to-network security: IPSec can be used to secure all traffic between two subnets in a private network.

End-to-end security: IPSec can provide secure communication between two specific devices or applications.

IKE

Transport or Tunnnel Mode

IPSec has two different modes for how much of the packet to encrypt:

1. Transport Mode: This mode encrypts only each packet’s data portion (payload), leaving the header untouched. It's typically used for end-to-end communications, such as connecting a client to a server.
2. Tunnel Mode: This mode encrypts the entire IP packet and wraps it inside another IP packet. This mode is useful for network-to-network communications (like in VPNs), as it allows a network device to act on behalf of another on a network.

IKE (Internet Key Exchange) v1

There are two types of key exchange within the IKE (Internet Key Exchange) protocol, which is used in IPsec VPNs to establish security associations (SA) and secret keys.

Only IKE version 1 has these two modes.

Main Mode

Main Mode is the standard method of IKE phase 1 negotiation. It consists of six messages:

1. The initiator sends encryption and hash capabilities to the responder.
2. The responder sends its encryption and hash capabilities to the initiator.
3. The initiator sends a pseudorandom number and its ID to the responder.
4. The responder sends a pseudorandom number and its ID to the initiator.
5. The initiator sends a message to authenticate itself.
6. The responder sends a message to authenticate itself.

Aggressive Mode

Aggressive Mode is a quicker method of IKE phase 1 negotiation. It only consists of three messages:

1. The initiator sends encryption and hash capabilities, a pseudorandom number, and its ID to the responder.
2. The responder sends a pseudorandom number, its ID, and an authentication message to the initiator.
3. The initiator sends an authentication message to the responder.

When to use each method?

Main Mode should be used when both ends of the VPN tunnel are static and identity protection is required. It is considered more secure because it provides identity protection during the negotiation process by encrypting the identities of the participating nodes.

Aggressive Mode might be necessary when one end of the VPN is dynamic (like a remote user or branch office with a dynamic IP address), as it establishes the VPN connection without knowing the remote IP address beforehand. This will speed up the initial connection time, but it is less secure.

Basically, when using Aggressive Mode, the information that can be seen is only the public facing IPs of the initiator and the responder. IKE SA details can also be seen.

This is really only an issue if the VPN used internal, private IPs or names that provide information about the internal network, like internalserver.company.com.

In the real world, I can’t imagine Aggressive Mode being much faster than Main unless the tunnel is established very frequently or the network latency is insane.

I think the main reason to use Aggressive mode would be if the responder’s IP address is unknown or if it changes frequently

IKEv2

IKEv2 is the newer version of the protocol. It simplifies the protocol exchanges, improves the negotiation efficiency, and provides better support for NAT-Traversal (NAT-T) and mobility.

IKEv2 does not have an explicit "aggressive" or "main" mode. Instead, it uses two exchanges (IKE_SA_INIT and IKE_AUTH) to accomplish what IKEv1 does in the "main" mode.

Benefits of version 2:

• IKEv2 simplifies the negotiation process, reducing the latency to establish a tunnel and making the process more efficient.
• IKEv2 handles NAT-Traversal better
• IKEv2 supports MOBIKE, which allows a VPN to stay connected even if the client changes networks or moves between different network interfaces, like between Wi-Fi and mobile data.

Authentication Header (AH) & Encapsulating Security Payload (ESP)

After the IKE SAs are established, the IKE protocol is used to negotiate additional SAs that are used by the IPsec AH and ESP protocols. These SAs contain the parameters needed for AH and ESP to secure the actual data traffic.

In both AH and ESP, the SAs define:

• Which encryption algorithm (like AES, DES, etc.) and key to use (for ESP),
• Which integrity check algorithm (like SHA, MD5, etc.) and key to use (for both AH and ESP),
• The lifetime of the keys (after which a new SA needs to be negotiated),
• How to handle replay attacks (anti-replay settings),
• And other necessary parameters for secure communication.

Establishing an IPsec VPN

Phase 1

This phase is also called the IKE phase.

Two VPN peers will establish a secure authenticated channel between themselves.

This is done by using the IKE protocol.

The peers will agree on the parameters for the IKE security association:

• Encryption algorithm
• Hash algorithm
• Diffie-Hellman group
• Authentication method
• Security Association Lifetime

The Diffie-Hellman algorithm is used to to generate a public-private key pair that can be exchanged over an unencrypted channel, but can be used to generate a shared secret.

The peers authenticate each other to confirm each other’s identity, usually done through digital certificates or pre-shared keys.

RESULT = IKE SA for traffic from Peer A to Peer B, and a separate IKE SA for the traffic from Peer B to Peer A.

Phase 2

The peers use the secure IKE SA from Phase 1 to negotiate the IPsec SAs.

This phase is also called the IPsec phase or the Quick mode in IKEv1.

Negotiation of IPsec policy: The peers agree on the parameters for the IPsec SA, such as the encryption algorithm, the hash algorithm, the mode of operation (tunnel mode or transport mode), and the SA lifetime.

Creation of IPsec SAs: The peers create two unidirectional IPsec SAs per VPN tunnel (one for each direction of traffic flow).

Optional additional Diffie-Hellman key exchange: Different key for IPsec SA than IKE SA. Optional. Provides additional security.

RESULT = IPsec tunnel established. Data can be encrypted and encapsulated according to the IPsec SA. The receiving peer decrypts and de-encapsulates the data according to the same IPsec SA parameters.

Additional details:

Initial Contact

Before the negotiation starts, the initiator sends an initial contact to the responder to check if it's available and willing to establish a VPN.

Proposal and Selection

Each phase of negotiation involves the peers sending a list of proposals for the parameters they support, and then selecting the best match. This can involve multiple rounds of messages if the first proposal is not accepted.

Nonces

Each phase of negotiation involves the peers generating and exchanging nonces - random numbers used once - to protect against replay attacks.

Key Derivation

The shared secret key generated via the Diffie-Hellman exchange is not used directly. Instead, it's used as input to a key derivation function to generate separate keys for different purposes.

Cookies

In IKEv2, the peers use cookies - random numbers chosen by the responder and included in the messages - as a protection against denial-of-service (DoS) and distributed denial-of-service (DDoS) attacks.

Lifetime and Renegotiation

Each SA has a lifetime after which it expires. Before it expires, the peers must renegotiate a new SA to replace it, repeating the relevant phase of negotiation.

Cryptographic Algorithms

Most common algorithms:

Encryption - DES, 3DES, AES (128,192,256 bit), AES-GCM

Integrity - MD5, SHA-1, SHA-2 (256, 384, 512 bit)

Key exchange - Diffie-Hellman Groups (different bits) including elliptic curve

Authentication - Pre-Shared Key (PSK), Digital Signature (RSA, ECDSA)

Troubleshooting?

The tunnel does not establish correctly?

1. Mismatch in configuration between two endpoints, Phase 1 or Phase 2 parameters, pre-shared keys or network subnets may be wrong.
2. Check the connectivity between the two endpoints; maybe a router or firewall is blocking the path?
3. Check logs on VPN gateway
4. The Security Associations can be captured in a packet capture during the main mode negotiation. This will show you the encryption algorithms and hash and authentication methods. Other packets will be encrypted.

The tunnel establishes but no traffic is flowing?

1. Verify that the IPsec policies (also called crypto map entries in some systems) are correctly configured to match the traffic you want to protect. (Policy routes).
2. Ensure that traffic behind the VPN gateway can reach the VPN in order to be sent down the tunnel
3. Check that NAT or a FW is not interfering with outbound traffic

The tunnel drops intermittently?

1. Check the SA liftetime, if the setting are expiring too often then the tunnel will have to renegotiate each time which can cause outages.
2. Check keepalives. If there’s no traffic the tunnel will stay online, if keepalive is disabled the tunnel may drop and then take time to reconfigure causing an “outage”.

Is the tunnel slow?

1. IPsec will add overhead due to encryption, a less intensive encryption algorithm might be better to use if performance is an issue.
2. Additional overhead might also cause issues with MTU and packet fragmentation. Best thing to do is lower MTU size or enable “Path MTU Discovery”.

Path MTU Discovery works by sending packets with a “Don’t Fragment (DF)” bit set in an IP header. If an intermediate router has a smaller MTU then it will drop the packet and send back an ICMP reply. The sender then reduces its MTU and tries again until communication is successful.