[ACP-DC Training #14] VXLAN

Why VXLAN?

The basis of VXLAN functionality is RFC 7348 As a standard, it provides an alternative that can replace the existing traditional VLAN concept.

1. Overcoming the limit of the number of VLANs (ID scalability)

Traditional VLANs are inside the 802.1Q header 12-bit Use numeric tags.
This is the maximum 4,094This means that you can only create different VLANs.

It is sufficient for a typical corporate environment, but with numerous customers (tenants) occupying it Multi-tenant data centerIn this area, this resource can be depleted quickly.

exampleLet's assume that a hosting provider uses 2 to 3 VLANs per server.

In this case, even if there are only about 2,000 servers, the available VLAN IDs will run out.
If a specific server requires more VLANs, the number of servers that can be accommodated will be significantly reduced.

2. Ease of deployment and maintenance

VXLAN is Edge devices of a transport networkYou just need to distribute it only.

Intermediate devices within the transmission network do not need to know VXLAN and simply need to perform standard L3 (Layer 3) routing (IP forwarding).
This means that you can expand the virtual network without touching the entire network.

3. Resolving the issue of excessive scaling of L2 broadcast domains

Another problem with traditional VLANs is The broadcast domain can become too large.is the point.
In a data center environment, many switches are connected to hypervisor servers that host virtual machines (VMs).

• Problem situationIt is difficult for network administrators to predict which physical servers and switches a specific virtual machine will run on.
  To prepare for this uncertainty, all server VLANs are also configured on all physical switches.
  This is because it ensures that communication is not interrupted regardless of which server the VM moves to (e.g., via vMotion).
• side effectIf you use this method in a large-scale data center with hundreds of switches massive broadcast domainThis is generated.
  There is a risk that unnecessary broadcast traffic will surge, security will become vulnerable, and the outage scope of network failures or errors will become too wide.

VXLAN technology provides a solution to these limitations of existing VLANs.

💡 Key Solution (Deep Dive)

• 12-bit vs 24-bit:
  While VLANs can only be created up to 4,094 using 12 bits, VXLAN uses a 24-bit identifier called VNI (VXLAN Network Identifier).
  Theoretically, about 16 million (2²⁴It can create a virtual network of ) and completely solves the ID exhaustion problem.

• L2 over L3 (tunneling)The biggest feature of VXLAN is “Putting an L2 packet inside an L3 packet (UDP)”no see.
  To use an analogy, it is like loading a 'passenger car named L2' onto a 'large truck named L3' and driving on a highway (IP network).
  By doing this, the intermediate routers only need to look at the truck (L3) and deliver it to the destination, and they do not need to worry about what the passenger car (L2) inside the truck is.

VXLAN Packet Structure

The core of VXLAN is the original Layer 2 (L2) frame Encapsulation inside a UDP/IP packetIt is to do.

In this process, containing a 24-bit VXLAN ID 8-byte VXLAN headeris added.

• Overhead occursIn this encapsulation process, the total 50 bytes of additional overheadIt occurs.

VXLAN header (8 bytes) + UDP header (8 bytes) + IP header (20 bytes) + External Ethernet header (14 bytes) = 50 bytes

• The Importance of MTU Setting: In order for the original L2 frame to carry a 1,500-byte payload intact, the IP infrastructure (underlay) wrapping it... The MTU must be at least 1,550 bytes.It must be set to.
  Otherwise, packets may be fragmented or lost.

Key Features of VXLAN (Operation)

VXLAN has the following features.

• Data Center Optimization: It is primarily used in data centers, but it is also used in campus networks to tunnel guest traffic to a gateway.
• IP-based L2 overlayIt uses the “MAC-in-UDP” format. In other words, it encloses L2 traffic within IP datagrams for transmission.
• Utilizing existing IP infrastructureOnce traffic is encapsulated, it is built within the data center or between data centers Use the existing IP routing infrastructure as isYou can. Thanks to this, you can enjoy all the redundancy, resilience, and load-sharing capabilities of IP networks.
• Overwhelming number of IDsSince 24-bit values are used, approximately 16 million (2²⁴It can have a VXLAN ID of ).

Key Concepts: VXLAN and VNI

The example topology in the figure above shows a Transport IP Network composed of multiple switches interconnected in a Routed Environment.
Multiple physical server hosts are connected to this infrastructure.

The key point of this example is that the server itself does not have VXLAN settings. Instead, it works as follows:

• Server sideVirtual machines (VMs) in the traditional way VLAN IDSends traffic including.
• Switch side (VTEP): The switch is this Mapping VLAN IDs to VXLAN IDs (VNIs)It performs the role and operates as VTEP.

In addition, this example shows that two IP backbone subnets are displayed.

• The 10.1.1.0/24 subnet contains two physical ESX servers named ESX-11 and ESX-12.
• There are also two hosts named ESX-21 and ESX-22 in the 10.1.2.0/24 subnet.

VNI Mapping Details

• ESX-11 server: Hosts VMs 1, 3, and 11.
• Switch settings:

• The traffic coming from VM 1 and VM 3 is VXLAN 1001It is set to be bound to.
• The traffic coming from VM 11 is VNI 2001It has been assigned to.
• Similar mapping settings are applied to other switches as well.

Operation result

The switch encapsulates the L2 (Layer 2) frame into an IP packet containing appropriate VNI information. The result is as follows:

• Virtual Network 1001: Scattered across different physical subnets VMs 1 through 5functionally Same L2 multipoint subnetYou will be in.
• Virtual Network 2001: It becomes an independent network including VM hosts 11, 12, and 13.

Key Point: VXLAN 1001 and 2001 operate like traditional classic VLANs without internal Layer 3 IP interfaces.
These are Complete isolationIt is configured so that VMs within each VXLAN can communicate with each other, but they cannot communicate with any device outside their VXLAN.

• Ignoring physical locationPhysically ESX-11과 ESX-21Although it is beyond this different router (different subnet),
  Thanks to VXLAN technology, the VMs on it feel as if they are plugged into the switch next to it.
• VNI = Virtual Network NameIt is easy to think of VNI as a 24-bit name tag that indicates "we are on the same team" on this vast ocean of IP networks.

Key Concept: VTEP (VXLAN Tunnel Endpoint)

The VXLAN Tunnel Endpoint (VTEP) is the entry point into the VXLAN world. Entry Point While performing its role, it interacts with other VTEPs to provide network connectivity.

Key Roles of VTEP

In a data center topology, each physical switch is assigned a role called VTEP and performs the following core functions:

1. Encapsulation: The leaf switch (VTEP) receives standard Ethernet frames from the server connected to it.
   After that, this frame is wrapped (encapsulated) into an IP/UDP packet along with a VXLAN header and sent to the destination through the tunnel.
2. Decapsulation: The destination VTEP receiving the packet unpacks (decapsulates) the incoming IP packet and extracts the original Ethernet frame contained within. It then delivers this to the final destination device (VM, etc.).

Figurative supplementary explanation: Highway on-ramp

Let's think of VTEP as an on-ramp on a highway.

• It is easy to understand this as the process where a car (frame) traveling on a regular road (local network) receives a pass (VXLAN header) at a tollgate (VTEP) to enter a highway (VXLAN tunnel) and is loaded onto a truck (IP packet).
• Conversely, upon arriving at the destination, it also serves as an exit to get off the truck and return to the regular road.

Who becomes a VTEP?: Primarily directly connected to the server Leaf switchIt performs the VTEP role.
The Spine switch in the middle acts solely as a delivery person, guiding the way based only on the attached IP address, regardless of whether the packet is VXLAN or not.

Hardware processingSince this encapsulation and decapsulation process must occur very quickly, modern switches such as AOS-CX handle this in hardware on a dedicated chipset (ASIC) to prevent performance degradation.

Key Concept: VXLAN Tunnel

When the two VTEPs start communicating with each other, in between VXLAN tunnelThis is formed.

This tunnel formation is essential to communicate with all destination VTEPs belonging to the same VXLAN.

• Tunnel characteristics: A tunnel is merely a simple transmission mechanism for passing through an IP network.
• The role of VNI: Multiple VXLAN IDs (VNIs) can pass through a single physical tunnel simultaneously.
  Since each packet contains unique VNI information, the receiving VTEP can look at it and classify the data into the appropriate virtual network to send.

Multi-destination traffic forwarding (BUM Traffic processing)

There are two methods for forwarding broadcast, multicast, and unknown unicast (BUM) traffic originating within the VXLAN through the underlying IP network (underlay).

1. Head-end Replication

This is a method where the source VTEP directly copies the packet and sends it to each destination VTEP one-to-one (Unicast).

• How it works: When Switch S1 receives a broadcast frame, it encapsulates it into VXLAN, creating one unicast packet for the destination S2 and one unicast packet for S3, and transmits them a total of two times.
• merit: No complex multicast configuration is required for the underlay infrastructure Deployment is very simpledo.
• disadvantage: As the number of VTEPs connected to a virtual network increases, the switch must generate a massive amount of replication packets every time a broadcast is sent, so Scalability issueThis may occur.

2. Multicasting

This is a method that maximizes efficiency by utilizing the multicast function of the underlying IP network.

• How it works: Source VTEP (S1) has a broadcast frame Encapsulate only onceThen, send it to a specific multicast address (e.g., 239.1.1.1). Other VTEPs (S2, S3) belonging to the corresponding VNI join this group in advance via IGMP.
• merit: Since VTEP only needs to send the packet exactly once regardless of the number of hosts Significantly save network resourcesIt has excellent scalability.
• disadvantage: The implementation becomes complex because the transmission network (underlay) must be configured separately to fully support multicast.

today The Aruba AOS-CX switch does not support this multicast method.
It is much more intuitive and stable because it primarily uses the unicast replication (headend replication) method.

Communication between Software VTEPs (Software VTEP to Software VTEP)

VXLAN tunnels between software VTEPs provide a communication environment for servers and services inside the virtualization server.

In this process, the underlying physical IP network (underlay) merely acts as a conduit to unicast VXLAN packets between virtualization servers and does not participate in the internal operations of VXLAN at all.

Key Features

• VM Connection Between Hypervisors: The physical hypervisor server itself performs the VTEP function to provide connectivity between virtual machines (VMs).
• Additional software requirements: Not all hypervisors have VTEP functionality built-in by default. To implement this, a separate network virtualization solution or software is usually required.
• Support Solutions: today OpenStackIna VMware NSXPlatforms such as [list of platforms] are equipped with these software-based VXLAN implementation capabilities.

Not all hypervisors support VTEP functionality. Additional software may be required.

• Change in the role of physical switches: If you use the software VTEP method, the physical switch (e.g., AOS-CX) does not need to know the VXLAN settings at all.
  The switch simply needs to forward standard IP packets from the server to the destination server.
• Subject of management: In this case, the initiative for network management belongs not to the physical network team. Virtualization Operations TeamIt will proceed to.
• maximum: Not all hypervisors fully support this feature, and in large-scale environments, server CPUs may consume resources for encapsulation/decapsulation operations, which could lead to reduced performance efficiency.

Communication between Hardware VTEPs (Hardware VTEP to Hardware VTEP)

This method of creating VXLAN tunnels between physical switches offers several clear advantages, such as the following.

Key Benefits

• Hypervisor Offloading The complex computational process of encapsulating and decapsulating VXLAN packets is handled by the switch's dedicated hardware chipset (ASIC) rather than the server's CPU. This allows the server to focus solely on service computations, thereby improving overall system performance.
• Integrated connection of virtual and physical devices: Not only virtual machines (VMs), but also general physical servers, storage equipment, and others that are completely unfamiliar with VXLAN functionality can be easily added to the VXLAN network.
• Separation of Universality and Management: It allows you to enjoy the benefits of VXLAN even in older hypervisor environments that lack VXLAN capabilities.

Even in situations where it is difficult to deploy VXLAN to devices managed by the server team or permissions are separated, the network team can independently build and manage overlay networks at the switch level.

Performance difference: Software VTEP shares server resources, but hardware VTEP uses a 'dedicated accelerator' designed inside the switch.
In data centers with large-scale traffic, this performance difference is critical.

Transparency It operates so transparently that servers or VMs do not even know they belong to VXLAN.
This is because the server simply throws an L2 frame as usual, and the switch automatically sends it through the tunnel.

Communication between software VTEP and hardware VTEP

You can create a VXLAN tunnel between a software VTEP (hypervisor) and a hardware VTEP (physical switch) to provide connectivity between virtual machines (VMs) and physical network services (routers, firewalls, etc.).

Key Features and Applications

• Combination of virtual and physical services: VMs inside VXLAN are physically Firewall, WAN router Or provides a path to access network services.
• Bare Metal Server Connection: Non-virtualized standard physical server (Bare Metal ServerThis is an essential method when VMs inside ) and VXLAN need to communicate with each other.
• Support Solutions: today Aruba AOS-CX Switch와 VMware NSX The environment fully supports such hybrid configuration features.

Summary of VXLAN Implementation Method Comparison

Let's compare the three scenarios we have learned so far at a glance.

Implementation method	Tunnel start/end point	Main uses
Software to Software	Hypervisor ↔ Hypervisor	Pure virtual communication between VMs
Hardware to Hardware	Physical switch ↔ Physical switch	Connection between physical devices or server load reduction (Offloading)
Software to Hardware	Hypervisor ↔ Physical Switch	Integrated connection between VMs and physical servers/security equipment

Ultimately, the true power of VXLAN is “Inclusiveness that binds equipment into a single network, regardless of whether it is physical or virtual”is in.
In particular, AOS-CX plays the role of a powerful hardware VTEP in such hybrid environments, completing the connectivity of the entire data center.

VSX Logical VTEP

VSX is a virtualization technology that makes two physical switches appear as a single device (L2) from the server's perspective and as independent devices (L3) from the core network's perspective.

1. Operating Principle: A smart design that is different on the inside and out

• Server side (MC-LAG): Two switches are connected to the server via MC-LAG (Multi-Chassis Link Consolidation).
  The server recognizes the two switches as a single switch, and through this, Active-Active Enjoy load balancing and path redundancy in mode.
• Core Side (Control Plane): The two switches each give different VTEPs or spine switches Independent BGP routing objectsIt operates as such. Control packets are processed individually to increase stability.
• Data Path: When carrying actual data, the two switches Single Logical VTEPIt works as.

2. Virtualization via Common IP (Anycast IP)

How can two switches appear to the outside with a single VTEP address?

• Common Loopback IP: On the loopback interface of the two switches Same IP addressSets.
• Source IP Settings: When configuring the VXLAN interface, this common IP source-ipDesignate as.
• Using ECMP: The spine switch recognizes that there are two paths to this common IP (VSX Primary and Secondary).
  Therefore, traffic is evenly distributed to the two switches via ECMP (Equal-Cost Multi-Path).

Data flow and synchronization

1. Traffic reception: When the external VTEP sends data to the host behind the VSX pair, set the destination address to this 'common logical VTEP IP'.
2. Arrival and Processing: The packet travels through the underlay network and arrives at one of the VSX pairs.
   VSX synchronizes MAC addresses and ARP information between the two switches in real time, so it can accurately deliver packets to the downstream host regardless of which switch they enter.
3. Asymmetric path support: Even if the host sends a packet out through switch 1, the response packet can come in through switch 2.
   Thanks to VSX synchronization, communication is processed normally without interruption even on these asymmetric paths.

In other words, the VSX-based logical VTEP configuration is “Strong redundancy”과 “Efficient bandwidth usage”This is a design that kills two birds with one stone.
Since this is an architecture that is essential when building a data center using AOS-CX equipment, utilizing a common loopback IP source-ip You must familiarize yourself with the setup method.

Connection from VXLAN to the physical network

Communication between internal VXLAN devices (Intra-VXLAN) alone is not sufficient.

If a VM wants to access the Internet or another subnet, just like in a traditional VLAN Default GatewayJust as an L3 IP gateway is required, an L3 IP gateway must also exist within the VXLAN.

Key Solutions

1. VM-Based IP Routing: This is a method of directly deploying a virtual machine with routing capabilities inside an ESX host.
2. VMware NSX: We provide routing services within the host using VMware's proprietary products.
3. Hardware-based Gateway: Aruba CX 8325Switches like this act as a bridge connecting VXLAN and standard VLAN at the hardware level.

---

VM-based IP Routing

This method involves creating a virtual machine (VM) with routing software installed and using it as a 'virtual router' to connect the two worlds.

Operating principle

structure:

ESX-11 Inside the server VM2Creates a virtual router named. This VM has two virtual NICs (vNICs).

• vNIC 1 (G1/0): Connect to VXLAN 1001 network (IP: 192.168.1.1)
• vNIC 2 (G2/0): Connected to a traditional VLAN 102 network (IP: 192.168.2.1)

Traffic flow:

1. in VXLAN 1001 VM1Its gateway when going outside 192.168.1.1(VM2Sends a packet to ).
2. virtual router VM2It receives this packet, performs L3 routing, attaches a VLAN 102 tag, and sends it out to the external switch through the physical NIC.
3. The external switch recognizes and processes it as normal VLAN traffic.

Advantages and Features

• pliability: Virtual Router (VM2) does not need to be fixed to a specific physical server.
  vMotionEven if you move to another server through it, the logical routing structure does not change.
• Versatility: It can be implemented on any operating system with routing capabilities (such as Linux).

From VXLAN to Physical Network: VMware NSX

VMware NSXThe solution using it has a structure very similar to virtual machine-based routing.

However, the biggest difference is that instead of using a standard OS, Dedicated virtual appliance provided by VMwareThe point is that it is distributed and used.

Key Features and Functions

• Integrated Service Provision: NSX provides not only simple IP routing but also the following advanced network services at the virtual layer.
  • Routing Support for communication between different virtual networks.
  • Filtering and Firewall: Application of micro-segmentation security policies at the virtual machine level.
  • Address Translation (NAT): Address translation between internal virtual IP and external physical IP.
• Operating principle: The basic operation is the same as the VM-based routing example.
  One virtual interface is connected to the VXLAN overlay, and the other to the physical VLAN (vSwitch), serving as a gateway between the two worlds.

---

Through this course, we will overcome the limitations of physical networks VXLANWe have examined the core principles and architecture of.

VXLAN, which emerged to solve the 4,094 ID constraint and broadcast domain expansion issues of traditional VLANs, layers a flexible L2 virtual tunnel (overlay) on top of a robust L3 IP routing infrastructure (underlay). ‘'MAC-in-UDP' encapsulation Through technology, up to 16 million independent virtual networks are implemented. In particular, the switch hardware directly handles encapsulation. Hardware VTEP, combining two switches into a single logical tunnel endpoint to maximize non-disruptive availability and ECMP load balancing VSX Logical VTEP, and various seamless connections between virtual environments and physical infrastructure Gateway SolutionIt is a core pillar of modern cloud data center design.

By combining the virtualization wings of VXLAN learned today with the spine-leaf fabric we built earlier, we are now able to design an agile and highly scalable data center network that remains stable regardless of workload shifts.