In this article, we will discuss about IPSec, its phases, different modes and its working. We will also configure IPSec with slight variations. Let’s get started:
What is IPsec?
IPsec, short for Internet Protocol Security, is a suite of protocols used to secure internet communications at the IP layer. It provides cryptographic security services for data integrity, confidentiality, and authentication, ensuring secure communication between devices over a public network like the internet.
Now look at the definition of following terms:
· Confidentiality: Encryption of our data so that only intended user or recipient will able to read or decrypt it.
· Integrity: We want to make sure that nobody changes the data in our packets.
· Authentication: The sender and receiver will authenticate each other to make sure that we are really talking with the device we intend to.
· Anti-replay: Even if a packet is encrypted and authenticated, an attacker could try to capture these packets and send them again. By using sequence numbers, IPsec will not transmit any duplicate packets.

IPsec can be used on many different devices, it is used on routers, firewalls, hosts and servers. Here are some examples how you can use it:
· Between two routers to create a site-to-site VPN that “bridges” two LANs together.
· Between a firewall and windows host for remote access VPN.
· Between two Linux servers to protect an insecure protocol like telnet.
Before we can protect any IP packets, we need two IPsec peers that build the IPsec tunnel.
To establish an IPsec tunnel, we use a protocol called IKE (Internet Key Exchange).
Now what is IKE (Internet Key Exchange)?
Internet Key Exchange (IKE) is a secure key management protocol that is used to set up a secure, authenticated communications channel between two devices.
There are two phases to build an IPsec tunnel:
· IKE phase 1:
In IKE phase 1, two peers will negotiate about the encryption, authentication, hashing and other protocols that they want to use and some other parameters that are required. In this phase, an ISAKMP (Internet Security Association and Key Management Protocol) session is established. This is also called the ISAKMP tunnel or IKE phase 1 tunnel.
· IKE phase 2: (We will discuss about it in a while.)
The collection of parameters that the two devices will use is called a SA (Security Association).
As per RFC 2408, A Security Association (SA) is a relationship between
two or more entities that describes how the entities will utilize security
services to communicate securely. This relationship is represented by a set
of information that can be considered a contract between the entities.
The information must be agreed upon and shared between all the entities.
Before going into more details, let us clear what ISAKMP and IKE are?
ISAKMP (Internet Security Association and Key Management Protocol) and IKE (Internet Key Exchange) are both protocols used in the establishment of Virtual Private Networks (VPNs) to ensure secure communication over the internet. Here’s a brief overview of each:
1. ISAKMP (Internet Security Association and Key Management Protocol):
Purpose: ISAKMP is a framework for establishing and negotiating security associations (SAs) and cryptographic keys in a network.
Functionality: It defines the procedures for authenticating the communicating parties, establishing, maintaining, modifying, and terminating SAs.
Features: ISAKMP operates at the network layer (Layer 3) and provides a set of standards for key exchange, typically using the Oakley and SKEME key exchange protocols.
2. IKE (Internet Key Exchange):
Purpose: IKE is a specific implementation of the ISAKMP framework, and it’s used to set up security associations (SAs) in the IPSec protocol suite.
Functionality: IKE performs key management by negotiating the SAs and generating the required cryptographic keys. It uses a combination of ISAKMP protocols, such as Oakley and SKEME.
Features: IKE handles the exchange of key generation and authentication information between devices, ensuring that the secure communication channels are properly established.
Phases: IKE has two main phases — Phase 1 and Phase 2. Phase 1 establishes a secure channel for further negotiations, and Phase 2 establishes the actual secure data transfer.
Key Differences:
Scope: ISAKMP is a broader framework that encompasses the entire process of security association and key management. IKE is a specific implementation of ISAKMP tailored for IPSec.
Implementation: ISAKMP can be used with various security protocols, while IKE is specifically designed for IPSec.
Phases: While ISAKMP is generally considered as a whole, IKE has distinct Phase 1 and Phase 2 processes for negotiation and data transfer, respectively.
Conclusion:
ISAKMP is a framework for dynamically establishing security associations and cryptographic keys in an Internet environment. This framework defines a set of message flows (exchanges) and message formats (payloads). ISAKMP defines a generic payload for key exchange information. This enables the ISAKMP protocol to manage cryptographic keys independent of the key exchange protocol that is used to generate them.
IKE augments the ISAKMP protocol to facilitate the creation of authenticated keying material. IKE defines how keying material is generated. The exchanges that are defined by ISAKMP require authentication to take place, but they do not specify how authentication is to be performed. IKE defines how authentication is to be performed.
Now you will have a better understanding of ISAKMP and IKE. Let’s continue the remaining discussion:
Here’s an example of two routers that have established the IKE phase 1 tunnel:

The IKE phase 1 tunnel is only used for management traffic like keepalives.. We use this tunnel as a secure method to establish the second tunnel called the IKE phase 2 tunnel or IPsec tunnel.
Here’s a picture of our two routers that completed IKE phase 2:

Once IKE phase 2 is completed, we have an IKE phase 2 tunnel (or IPsec tunnel) that we can use to protect our user data. This user data will be sent through the IKE phase 2 tunnel:

IKE builds the tunnels for us but it doesn’t authenticate or encrypt user data. We use two other protocols for this:
· AH (Authentication Header)
· ESP (Encapsulating Security Payload)
AH and ESP both offer authentication and integrity but only ESP supports encryption. Because of this, ESP is the most popular choice nowadays.
Both protocols (AH and ESP) support two different modes:
· Transport mode
· Tunnel mode
The main difference between the two is that with transport mode we will use the original IP header while in tunnel mode, we use a new IP header. Here’s an example for better understanding:

Transport mode is often between two devices that want to protect some insecure traffic (example: telnet traffic).
Tunnel mode is typically used for site-to-site VPNs where we need to encapsulate the original IP packet since these are mostly private IP addresses and can’t be routed on the Internet. I will explain these two modes in detail later in this article.
The entire process of IPsec consists of five steps:
- Initiation: Something has to trigger the creation of our tunnels. For example, when you configure IPsec on a router, you use an access-list to tell the router what data to protect. When the router receives something that matches the access-list, it will start the IKE process. It’s also possible to manually initiate the tunnel.
- IKE phase 1: We negotiate a security association to build the IKE phase 1 tunnel (ISAKMP tunnel).
- IKE phase 2: Within the IKE phase 1 tunnel, we build the IKE phase 2 tunnel (IPsec tunnel).
- Data transfer: We protect user data by sending it through the IKE phase 2 tunnel.
- Termination: When there is no user data to protect then the IPsec tunnel will be terminated after a while.
Now you have an idea of the basics of IPsec, let’s take a closer look at each of the different components.
IKE (Internet Key Exchange)
IKE (Internet Key Exchange) is one of the primary protocols for IPsec since it establishes the security association between two peers. There are two versions of IKE:
· IKEv1
· IKEv2
IKEv1 was introduced around 1998 and superseded by IKEv2 in 2005. There are some differences between the two versions:
· IKEv2 requires less bandwidth than IKEv1.
· IKEv2 supports EAP authentication (next to pre-shared keys and digital certificates).
· IKEv2 has built-in support for NAT traversal (required when your IPsec peer is behind a NAT router).
· IKEv2 has a built-in keep alive mechanism for tunnels.
The list with advantages goes on but for now, let’s focus on understanding IKE. As explained before, IKE uses two phases:
· IKE Phase 1
· IKE Phase 2
Let’s discuss what happens at each phase. Everything I explain below applies to IKEv1.
IKE Phase 1
The main purpose of IKE phase 1 is to establish a secure tunnel that we can use for IKE phase 2.
We can break down phase 1 in three simple steps:
Step 1: Negotiation
The peer that has traffic that should be protected will initiate the IKE phase 1 negotiation. The two peers will negotiate about the following items:
· Hashing: we use a hashing algorithm to verify the integrity, we use MD5 or SHA (SHA-1 or SHA-2) for this.
· Authentication: each peer has to prove who he is. Two commonly used options are a pre-shared key or digital certificates.
· DH (Diffie Hellman) group: the DH group determines the strength of the key that is used in the key exchange process. The higher group numbers are more secure but take longer to compute.
· Lifetime: how long does the IKE phase 1 tunnel stand up? the shorter the lifetime, the more secure it is because rebuilding it means we will also use new keying material. Each vendor uses a different lifetime; a common default value is 86400 seconds (1 day).
· Encryption: what algorithm do we use for encryption? For example: DES, 3DES or AES.
Step 2: DH Key Exchange
Once the negotiation has succeeded, the two peers will know what policy to use. They will now use the DH group that they negotiated to exchange keying material. The end result will be that both peers will have a shared key.
Step 3: Authentication
The last step is that the two peers will authenticate each other using the authentication method that they agreed upon on in the negotiation. When the authentication is successful, we have completed IKE phase 1. The end result is a IKE phase 1 tunnel (aka ISAKMP tunnel) which is bidirectional. This means that both peers can send and receive on this tunnel.
The three steps above can be completed using two different modes:
· Main mode
· Aggressive mode
Main mode uses six messages while aggressive mode only uses three messages. Main mode is considered more secure since identification is encrypted, aggressive mode does this in clear-text.
Our IKE phase 1 tunnel is now up and running and we are ready to continue with IKE phase 2.
The IKE phase 2 tunnel (IPsec tunnel) will be actually used to protect user data. There is only one mode to build the IKE phase 2 tunnel which is called quick mode.
Just like in IKE phase 1, our peers will negotiate about a number of items:
· Encapsulation: Do we use AH or ESP?
· Encapsulation Mode: Transport or tunnel mode?
· Encryption (Confidentiality): What encryption algorithm do we use? DES, 3DES or AES?
· Hashing (Integrity): What hashing algorithm do we use? MD5, SHA-1 or SHA-2?
· Authentication: what authentication algorithm do we use? Pre- Shared Keys (PSK) or Certificates or ECDSA?
· Lifetime: how long is the IKE phase 2 tunnel valid? When the tunnel is about to expire, we will refresh the keying material.
· (Optional) DH exchange: used for PFS (Perfect Forward Secrecy).
PFS is optional and forces the peers to run the DH exchange again to generate a new shared key in each IKE phase 2 quick mode.
Once IKE phase 2 has completed, we are finally ready to protect some user data. Let’s see how this is done…
IKEv2 doesn’t have a main or aggressive mode for phase 1 and there’s no quick mode in phase 2. It still has two phases though, phase 1 is called the IKE_SA_INIT and the second phase is called IKE_AUTH. Only four messages are required for the entire exchange.
IPsec Protocols
AH and/or ESP are the two protocols that we use to actually protect user data. Both of them can be used in transport or tunnel mode, let’s walk through all the possible options.
Authentication Header (AH) Protocol
AH offers authentication and integrity but it doesn’t offer any encryption. It protects the IP packet by calculating a hash value over almost all fields in the IP header. The fields it excludes are the ones that can be changed in transit (TTL and header checksum). Let’s start with transport mode…
Transport Mode:
Transport mode is simple; it just adds an AH header after the IP header. Here’s an example of an IP packet that carries some TCP traffic:

And here’s what that looks like in Wireshark:

Above you can see the AH header in between the IP header and ICMP header. This is a capture I took of a ping between two routers. You can see that AH uses 5 fields:
· Next Header: this identifies the next protocol, ICMP in our example.
· Length: this is the length of the AH header.
· SPI (Security Parameters Index): this is an 32-bit identifier so the receiver knows to which flow this packet belongs.
· Sequence: this is the sequence number that helps against replay attacks.
· ICV (Integrity Check Value): this is the calculated hash for the entire packet. The receiver also calculates a hash, when it’s not the same you know something is wrong.
Tunnel Mode:
With tunnel mode we add a new IP header on top of the original IP packet. This could be useful when you are using private IP addresses and you need to tunnel your traffic over the Internet. It’s possible with AH but it doesn’t offer encryption:

The entire IP packet will be authenticated. Here’s what it looks like in Wireshark:

Above you can see the new IP header, then the AH header and finally the original IP packet that carries some ICMP traffic.
One problem with AH is that it doesn’t play well with NAT / PAT. Fields in the IP header like TTL and the checksum are excluded by AH because it knows these will change. The IP addresses and port numbers however are included. If you change these with NAT, the ICV of AH fails.
Encapsulating Security Payload (ESP) Protocol
ESP is the more popular choice of the two since it allows you to encrypt IP traffic. We can use it in transport or tunnel mode, let’s look at both.
Transport Mode
When we use transport mode, we use the original IP header and insert an ESP header. Here’s what it looks like:

Above you can see that we add an ESP header and trailer. Our transport layer (TCP for example) and payload will be encrypted. It also offers authentication but unlike AH, it’s not for the entire IP packet. Here’s what it looks like in Wireshark:

Above you can see the original IP packet and that we are using ESP. The IP header is in clear text but everything else is encrypted.
Tunnel Mode:
How about ESP in tunnel mode? This is where we use a new IP header which is useful for site-to-site VPNs:

It’s similar to transport mode but we add a new header. The original IP header is now also encrypted.
Here’s what it looks like in Wireshark:

The output of the capture above is similar to what you have seen in transport mode. The only difference is that this is a new IP header, you don’t get to see the original IP header.
TOPOLOGY
GRE Over IPsec:
1. Tunnel Mode
2. Transport Mode
LAB Description:
This topology is implemented in tunnel mode and transport mode. Configurations and verifications are pasted for this lab and Wireshark packets are also captured for this lab.

To be Continued…….
References:
1. networklessons.com
2. https://datatracker.ietf.org/doc/html/rfc2408