IPSec Deep Dive

Today, we’re kicking off our series of posts with one about IPsec.

First of all — IPsec? What is it?

IPsec is a suite of protocols that provide security for Internet communications at the IP layer [1]. It has three main goals:

Authentication- Who sent the packet ?

Integrity -Was the packet modified in transit?

Confidentiality - Can anyone read the packet?

Anti-replay - Did I already received it?

One of the protocols that IPsec uses — and probably the most important — is called IKE [2]. The first version of IKE will be our main topic today. IKE stands for Internet Key Exchange. As the name suggests, the IKE protocol defines how negotiation between IPsec peers is carried out.

Although we often refer to IKE and ISAKMP interchangeably, IKE actually inherits capabilities from three different protocols:

ISAKMP: Provides a framework for authentication and key exchange but does not define them.

OAKLEY: Describes a series of key exchanges, called modes. It also details the services provided by each, such as perfect forward secrecy for keys, identity protection, and authentication.

SKEME: Describes a versatile key exchange technique that provides anonymity, repudiability, and quick key refreshment.

IKE operates in two main phases: Phase 1 and Phase 2. During Phase 1, a secure control tunnel is established between peers. Parameter negotiation can occur in one of two modes — Main Mode or Aggressive Mode. In Phase 2, a second tunnel is established for data traffic. This phase operates in a single mode, known as Quick Mode, and is often referred to as the IPsec Phase.

To really understand the details, we’re going to take a closer look at the IKE packet exchange. As always, Wireshark will be our best ally! For this analysis, we’ll use Main Mode for IKE Phase 1.

So, let’s dive in!

IKE phase 1

Packet #1:

The Responder selects the supported transforms from the options sent by the Initiator and also responds with its Vendor IDs. The Responder Vendor IDs indicate that it also supports DPD and NAT Traversal, hence those functionalities are enabled. Note that the Responder also includes its SPI value.

Packets #3 and #4:

Diffie–Hellman public keys, nonces (large random numbers), and NAT discovery hashes are exchanged between peers. After the fourth packet is sent, three different keys are generated, and subsequent packets are encrypted. These keys are:

      SKEYID_e - Used by ISAKMP to provide confidentiality for its messages.

      SKEYID_a - Used by ISKAMP SA to authenticate its messages.

      SKEYID_d - Used to calculate subsequent IPSEC keying material. 

Packets #5 and #6:

Those packets are encrypted using the SKEYID_e key. Although we can’t see it, each packet includes an Identity Payload (hostname or IP address) and a hash used for authentication purposes. Notice that at this point, the UDP port changes to 4500. This is because NAT Traversal is now enabled between peers, and ISAKMP packets are encapsulated. We’ll talk more about this technique in our next post.

IKE Phase 2 (Quick Mode/IPSEC Phase)

Packets #1 and #2:

Again, those packets are encrypted, and we cannot see their contents. At this stage, several parameters are exchanged:

• New SA Proposal: Sent by the initiator and selected by the responder. In this proposal, we also define whether we are going to use Encapsulating Security Payload (ESP) or Authentication Header (AH). These are two important protocols used by IPsec to provide confidentiality, authentication, integrity, and anti-replay protection.
• Initiator and Responder SPIs: The SPIs uniquely identify the unidirectional IPsec SAs. In this phase, rather than having a single bidirectional SPI, both peers will have locally unique inbound and outbound SPIs.
• Perfect Secrecy Forward (PFS): If enabled, a new Diffie Hellman material key is created.
• Proxy-IDs: Both peers identify the interesting traffic that are going to traverse the tunnel.
• The IPSEC mode: Tunnel or Transport.

Packet #3:

The last packet contains a hash sent by the Responder. It confirms the Responder’s liveness and also helps mitigate limited DDoS attacks.

So far, for IKE Phase 1, we’ve only talked about Main Mode - but what about Aggressive Mode?

When IKE Phase 1 operates in Aggressive Mode, it exchanges only three packets, compared to the six packets used in Main Mode. You might be thinking, “Finally! Fewer packets — troubleshooting should be much easier!”

But wait a minute…

Why don’t cloud providers - such as AWS - allow Aggressive Mode as a configuration option for their VPN services?

Because security is always the top priority. Let’s see why Aggressive Mode is considered insecure. In Aggressive Mode, the Pre-Shared Key (PSK) is sent together with the Diffie–Hellman key exchange payloads. All the values required to calculate the hash are present in the first Responder packet - and this packet cannot be encrypted, since the key exchange has not yet completed.

Therefore, an attacker who can capture the exchanged traffic can obtain the unencrypted hash and all the other values needed to calculate it (except for the PSK itself).

At this point, you’ve probably noticed that IPsec is a complex suite of protocols. To truly understand it, you need a solid grasp of the main components it relies on.

Today, we’ve introduced a few of those protocols and mechanisms — some of which you may not have encountered before — such as Dead Peer Detection (DPD), NAT Traversal, Diffie–Hellman, ESP, and AH.