VMware NSX-V - Replication Modes

When BUM (Broadcast, Unknown unicast and Multicast) traffic is sent out from a VM, the source VTEP must replicate this traffic out to the other VTEPs via VXLAN encapsulation.

NSX provides 3 modes of replication - Multicast, Unicast and Hybrid.

Multicast

Multicast replication utilizes L2 (IGMP) and L3 (PIM) multicast forwarding upon the underlay.

Based on Figure 1, let us walk through the steps:

1. BUM traffic is sent from the VM out to the VTEP.
2. The VTEP will then send the traffic out to the multicast group.
3. IGMP will then forward the traffic out to the other local VTEPs and also the router.
4. The router will then forward the multicast traffic (Layer3) to other routers via PIM. Traffic from the router is sent back out to the switch via IGMP.
5. IGMP will then be used to replicate the traffic out to the other VTEPs within the L2 segment.
6. After the VTEP has decapsulated the packet, the BUM traffic is passed up to the VM.

The advantage of Multicast replication is that the source VTEP only has to send the traffic once.The disadvantage, though, is that greater configuration is required within the underlay due to the requirement of IGMP and PIM.

Figure 1 - Multicast mode. ^[1]

Unicast

With Unicast mode, the ESXi hosts are divided into separate groups/VTEP segments, based on the VTEP subnets. A single ESXi host in each VTEP segment is elected as the UTEP (Unicast Tunnel End Point). The role of the UTEP is to receive traffic from remote VTEPs and then to forward the traffic onto the local VTEPs.

Based on Figure 2, let us walk through the steps:

1. BUM traffic is sent from the VM out to the VTEP.
2. The VTEP sends the traffic out multiple times (via unicast) to each of the local VTEPs, and also to the UTEP(s).
3. The UTEP sends the traffic out multiple times (via unicast) to each of the local VTEPs.
4. After the VTEP has decapsulated the packet, the BUM traffic is passed up to the VM.

The advantage with unicast mode is that no configuration is required on the underlay. However, the key disadvantage is that traffic is sent out multiple times from the source VTEP and also the UTEP to each of the local VTEPs.

Figure 2 - Unicast mode. ^[2]

Hybrid

Hybrid replication is a cross of both unicast and multicast replication. Hybrid mode leverages IGMP but negates the need for PIM. In addition, it also introduces a new type of proxy VTEP, the MTEP (Multicast Tunnel Endpoint).

Based on Figure 3, let us walk through the steps:

1. BUM traffic is sent from the VM out to the VTEP.
2. The VTEP will then send the traffic out to the multicast group. IGMP will forward the traffic to the local VTEPs.
3. The source also sends the traffic via unicast out to the MTEP of each remote VTEP segment.
4. The MTEP will then send the traffic out to the multicast group, allowing IGMP to forward to each of the local VTEPs. IGMP will then forward the traffic out to the other local VTEPs and also the router.
5. After the VTEP has decapsulated the packet, the BUM traffic is passed up to the VM.

The advantage with Hybrid mode is that PIM is longer required. However, traffic is still sent twice by the source VTEP. Once to the multicast group, and once to the MTEP via unicast.

Figure 3 - Hybrid Mode. ^[3]

ARP Suppression

In the scope of this article, we should mention ARP suppression, due it not being supported across all replication modes.

This is shown below:

So, What is ARP suppression?
ARP suppression is a feature that reduces the amount of broadcast traffic sent over the network.
At the point a VM boots up, and starts to send traffic, the local VTEP builds the IP to MAC address mapping. The VTEP then sends this information to NSX Controller.
This greatly helps reduce broadcast traffic, as when a VM sends an ARP request (broadcast) the VTEP intercepts the request and forwards it to the NSX controller. The NSX controller then responds to the request, which the host caches and passes onto the VM. Therefore preventing all hosts from receiving the ARP (request) broadcast.

References