Layer 5 Models

How Routers Handle ISAKMP Segments

When a router sends an ISAKMP segment (flowchart here):

• If a packet is getting out of the device is not encrypted, and it is an interesting traffic, and also if the outgoing port is configured with a crypto map command:
• ISAKMP initiates the negotiation for the first peer. It starts from the lowest crypto map sequence number and tries to negotiate with all peers in the list until the first success.
• If an IKE peer with the same IP is not configured, the initiator starts phase I negotiation.
• The initiator device sends out the first packet of negotiation by encapsulating all the ISAKMP policies that have been configured with.
• The responder device sends out the second message of negotiation by
• If match found, it encapsulates the matched policy.
• Otherwise, sends a packet to tell the peer to discontinue the negotiation.
• The initiator device sends out the third message:
• If no match, it deletes the IKE peer.
• Otherwise, it sends the third message which includes the nonce numbers and the key for the peer to calculate DH values.
• The responder device sends out the fourth message:
• Sends out its nonce numbers and the key to the peer
• The initiator device sends the fifth message:
• Encrypts the identity of the receiver with the DH key and sends it to the peer.
• The responder sends the sixth message:
• Encrypts the identity of the sender with DH key and sends it to the peer.
• Otherwise phase II:
• The initiator device encrypts the SA payload with the DH key calculated in the phase I, encapsulates it in an ISAKMP and sends it to the peer.
• The responder device sends the second message of phase II:
• If the match policy with sender found, it sends the encrypted matching policy to the peer.
• Otherwise, it sends a notification to the peer, drops the packet, and deletes the IKE peer.
• The initiator device:
• If the receiving device has sent back a matching policy, the initiator sends an ACK to the receiving device and successfully finishes phase II.
• Otherwise, it deletes the IKE peer.

When a router receives an ISAKMP segment:

• If the port, that is receiving the ISAKMP packet, is not configured with crypto map:
• It drop the packets and stops.
• If the port, that has received the ISAKMP packet, is configured with crypto map:
• It checks if an IKE peer exists with the source IP of the receiving packet.
• IKE phase I negotiation is already done, and it starts negotiating for IKE phase II.
• If an IKE peer does not exist with the same source IP of the receiving packet:
• The Initiator adds the peer to the IKE peer list and starts negotiating IKE phase I with the new peer.
• If the packet is the first IKE packet, the responder device processes the security associates payload to find a matching policy.
• If a key exists, that has been associated with the peer IP, and a matching policy found.
• It will continue with the negotiation by sending the accepting policy to the peer.
• Otherwise, it sends a packet and rejects the policy. It also deletes the associated IKE peer.
• If the packet is the second packet of ISAKMP main mode, the initiator processes the payload:
• If the payload is a notification, it processes the payload and deletes the IKE peer because there was not an existing policy or key.
• If the payload is a security associate payload which means that the peer has found a match with that policy the device:
• Encapsulates a nonce (g,p) payload and its key payload in an ISAKMP packet and sends it back to the peer.
• Calculates the public key (TA = g power a mod p). "a" is the private key for this peer.
• If the receiving packet is the third packet of the main mode negotiation, the responder processes the nonce and key payload.
• Get the nonce (g,p)
• Calculate its public key (TB = g power b mod p). "b" is the private key for this peer.
• The receiving device also encapsulates a nonce (g,p) payload and its key in an ISAKMP packet and sends it back to the peer.
• Calculates the DH shared secret K = TA power b mod p.
• If the packet is the fourth packet of the main mode negotiation that is receiving initiator processes the private key and nonce payload:
• Calculates the DH shared secret key K = TB power a mod p.
• Encrypts the peer IP with K and sends it to the peer
• If the packet is the fifth packet of the main mode negotiation responder:
• Gets the payload and identifies the identity of the peer by decrypting the encrypted msg.
• Encrypts the peer IP with K and sends it to the peer.
• Marks the IKE peer for phase II negotiation.
• If the packet is the sixth packet of the main mode negotiation initiator:
• Gets the payload and identifies the identity of the peer by decrypting the encrypted message.
• Marks the IKE peer for phase II negotiation.
• Initiator also starts phase II by encrypting (with the key) the existing SAs and sends it to the peer.
• IKE peer exists and marks with the phase II negotiation.
• If the received packet is the first packet of the quick mode negotiation the responder processes it and sends the second message:
• It gets the ISAKMP data and decrypts it with the key that has been calculated in phase I.
• Goes through all the security associates that the peer has sent.
• If it finds a match
• Generates outbound SAs.
• Encrypts and sends the accepting SA to the peer.
• Creates an IPsec peer.
• If no matching SA found,
• Sends a packet and denies continuing the negotiation.
• If the packet has received is the second message of the quick mode, the initiator processes it and sends the third message:
• It gets the ISAKMP data and decrypts it with they key that has been calculated in phase I.
• If SA has been accepted by the peer:
• Gets the accepted security associate that the peer has accepted.
• Generates outbound SAs.
• Creates IPsec peer.
• Encrypts and sends an ACK to the peer.
• If SA has not been accepted by the peer:
• Discontinues the negotiation.
• If the received packet is the third message of the quick mode, the responder gets the ACK and creates SAs and IPSec peers. IT marks the peer with phase II completed.

How PCs Handle ISAKMP Segments

If PC, which is an easy VPN client, has a request to connect to an easy VPN server (flowchart here):

• PC sends ISAKMP policies that it has been configured with plus nonce, a key, and identification payload.
• Server sends a reply back after getting the first packet and checking for AAA values.
• If the AAA authorization is not configured, it drops the packet.
• Otherwise, it continues with the IKE phase I negotiation:
• Either finds a matching policy
• Server sends the accepting policy back plus ID, nonce, and key payload back to the client.
• The IKE peer on the server enters XAUTH mode
• Otherwise
  • It does not find a matching policy, it drops the packet.
  • If the group name and group key, that the client has sent the request to, do not match or do not exist, it drops the packet.
• If the server has sent back its ID, Key and nonce with the matching policy.
• PC sends back an ACK.
• This ACK packet is encrypted with the DH shared key that has been calculated with nonce, and key of the received packet from the server.
• IKE peer on the PC enters XAUTH negotiation phase.
• Otherwise, it drops the packet and removes the PKE peer.
• Server sends its first XAUTH packet to the client:
• If server is configured with AAA authentication for the client it sends a prompt to the client and requests the client's username/password.
• Otherwise server authenticates the user and continues with the next message.
• The client sends username/password when it receives the prompt from the server.
• The server receives the username/password. It consults with either AAA server or its local configuration (based on the configuration):
• If match found:
• Authenticates the client
• Gets an IP from the IP pool and sends it to the client.
• Otherwise, it denies the client to access the server and sends a notification back to the client.
• The client receives the packet:
• If it is a notification, the client stops the negotiation.
• Otherwise, the client gets the IP, assigns the IP to its tunnel interface, and sends the last message of XAUTH mode, which is an ACK to the server.
• Server gets the ACK and:
• Creates a new route in its routing table to the tunnel IP address of the new client.
• Creates an interesting traffic for the client.
• Marks the IKE peer with XAUTH mode complete .
• Sends a packet to the client to notify the PC to start the phase II.
• Client receives the notification from the server and sends first phase II packet to the server. (for phase II please refer to router's phase II)

If PC which is an easy VPN client has a request to disconnect from an easy VPN server (flowchart here):

• The client sends an informational packet to the server and requests to disconnect.
• The server receives the request to disconnect from the client:
• Removes the static route to the client.
• Releases the borrowed IP to the pool.
• Removes the interesting traffic and its associated SA for the client from its interested traffic table.
• Sends an informational packet back to the client.
• The client receives the informational packet that sever has been disconnected:
• Removes the tunnel interface and disconnects.