• Part1: Underlay --> Part2: Tunnel --> You Are Here
Chapter: Troubleshooting BGP EVPN VXLAN
https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst9400/software/release/17-5/configuration_guide/vxlan/b_175_bgp_evpn_vxlan_9400_cg/troubleshooting_bgp_evpn_vxlan.html#id_131532
MP-BGP EVPN is a standards-based control plane that supports the distribution of attached host MAC and IP addresses using MP-BGP, namely, using the EVPN address family and MAC addresses treated as routing entries in BGP. As a control plane that is separate from the data plane, MP-BGP EVPN avoids flood-and-learn in the network, and the wide use of BGP as an external gateway protocol on the internet proves its ability to scale well with large deployments.
The following topology demonstrates how MP-BGP EVPN distributes route type 2 MAC/IP advertisement routes among VTEPs in the VXLAN, and minimizes ARP broadcast traffic required for PC A to learn PC B's MAC address when pc-A tries to connect to pc-D.
(According to Fortinet: https://docs.fortinet.com/document/fortigate/7.4.4/administration-guide/52499/vxlan-with-mp-bgp-evpn)
Create a full mesh of three iBGP peers.
BGP will use OSPF routes for connectivity
Leaf-1
router bgp 65001
bgp router-id 1.1.1.1
bgp log-neighbor-changes
no bgp default ipv4-unicast
neighbor 10.2.2.2 remote-as 65001
neighbor 10.2.2.2 update-source Loopback1
neighbor 10.3.3.3 remote-as 65001
neighbor 10.3.3.3 update-source Loopback1
!
address-family ipv4 !<--- No IP addresses, no Layer 3 routes
exit-address-family
!
! address-family ipv4 mvpn <--- Multicast routing not needed in BGP
!
address-family l2vpn evpn
neighbor 10.2.2.2 activate
neighbor 10.2.2.2 send-community both
neighbor 10.3.3.3 activate
neighbor 10.3.3.3 send-community both
exit-address-family
Leaf-2
router bgp 65001
bgp router-id 2.2.2.2
bgp log-neighbor-changes
no bgp default ipv4-unicast
neighbor 10.1.1.1 remote-as 65001
neighbor 10.1.1.1 update-source Loopback2
neighbor 10.3.3.3 remote-as 65001
neighbor 10.3.3.3 update-source Loopback2
!
address-family ipv4
exit-address-family
!
address-family l2vpn evpn
neighbor 10.1.1.1 activate
neighbor 10.1.1.1 send-community both
neighbor 10.3.3.3 activate
neighbor 10.3.3.3 send-community both
exit-address-family
Spine-3
router bgp 65001
bgp router-id 3.3.3.3
bgp log-neighbor-changes
no bgp default ipv4-unicast
neighbor 10.1.1.1 remote-as 65001
neighbor 10.1.1.1 update-source Loopback3
neighbor 10.2.2.2 remote-as 65001
neighbor 10.2.2.2 update-source Loopback3
!
address-family ipv4
exit-address-family
!
address-family l2vpn evpn
neighbor 10.1.1.1 activate
neighbor 10.1.1.1 send-community both
neighbor 10.2.2.2 activate
neighbor 10.2.2.2 send-community both
exit-address-family
It might be a little slow in the start... ??
C:\pc-A>ping 10.1.30.20
Pinging 10.1.30.20 with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
C:\pc-B>ping 10.1.30.10
Pinging 10.1.30.10 with 32 bytes of data:
Reply from 10.1.30.10: bytes=32 time=1ms TTL=128
Reply from 10.1.30.10: bytes=32 time=2ms TTL=128
Reply from 10.1.30.10: bytes=32 time=2ms TTL=128
C:\pc-A>ping 10.1.30.20
Pinging 10.1.20.10 with 32 bytes of data:
Reply from 10.1.20.10: bytes=32 time=2ms TTL=128
Reply from 10.1.20.10: bytes=32 time=1ms TTL=128
Reply from 10.1.20.10: bytes=32 time=2ms TTL=128
Leaf-2
show bgp l2vpn evpn summary
BGP router identifier 2.2.2.2, local AS number 65001 BGP table version is 1, main routing table version 1 Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 10.1.1.1 4 65001 5 5 1 0 0 00:02:07 3 10.3.3.3 4 65001 5 5 1 0 0 00:02:06 0
Source: Commands from "Troubleshooting EVPN/VXLAN on Catalyst Catalyst 9000 IOS-XE"
https://www.ciscolive.com/c/dam/r/ciscolive/emea/docs/2024/pdf/LTRTRS-2556.pdf Cisco Live 2024 LTRTRS-2556
Leaf-1 and Leaf-2
show l2vpn evpn mac ip
Leaf-1#show l2vpn evpn mac ip IP Address EVI VLAN MAC Address Next Hop(s) --------------------------------------- ----- ----- -------------- ----------- 10.1.30.10 30 10 a4bb.6da2.efbc Gi1/0/10:10 10.1.30.20 30 10 a4bb.6da2.e2a6 10.2.2.2
Leaf-2#show l2vpn evpn mac ip IP Address EVI VLAN MAC Address Next Hop(s) --------------------------------------- ----- ----- -------------- ----------- 10.1.30.10 30 20 a4bb.6da2.efbc 10.1.1.1 10.1.30.20 30 20 a4bb.6da2.e2a6 Gi1/0/10:20
Leaf-1 and Leaf-2
show bgp l2vpn evpn summary
Leaf-2# <--- output deleted for clarity ---> Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 10.1.1.1 4 65001 58 43 52 0 0 00:35:04 2 10.3.3.3 4 65001 40 43 52 0 0 00:34:24 0
L2RIB (L2 Routing Information Base)
show l2route evpn mac ip topology 30
NVE verification
show nve vni
NVE verification
show nve peers vni 12300
Look for "10.1.1.1" in the multicast-group 239.30.30.30
Leaf-2#show ip mroute
IP Multicast Routing Table
< - - Output omitted for clarity - - >(*, 239.30.30.30), 00:20:57/stopped, RP 10.3.3.3, flags: SJCx
Incoming interface: GigabitEthernet1/0/4, RPF nbr 10.0.4.3
Outgoing interface list:
Tunnel1, Forward/Sparse-Dense, 00:20:51/00:00:08
(10.1.1.1, 239.30.30.30), 00:02:49/00:00:10, flags: JTx
Incoming interface: GigabitEthernet1/0/4, RPF nbr 10.0.4.3
Outgoing interface list:
Tunnel1, Forward/Sparse-Dense, 00:02:49/00:00:10
(*, 224.0.1.40), 00:34:56/00:02:05, RP 10.3.3.3, flags: SJCL
Incoming interface: GigabitEthernet1/0/4, RPF nbr 10.0.4.3
Outgoing interface list:
Loopback2, Forward/Sparse, 00:34:54/00:02:05
Leaf-2#
From Cisco Live - not working in this example
NOT Working here! - show ip mroute 239.10.10.10 10.255.1.1 - nice to know
VTEP-1# show ip mroute 239.10.10.10 10.255.1.1 IP Multicast Routing Table Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected, L - Local, P - Pruned, R - RP-bit set, F - Register flag, T - SPT-bit set, J - Join SPT, M - MSDP created entry, E - Extranet, X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement, U - URD, I - Received Source Specific Host Report, Z - Multicast Tunnel, z - MDT-data group sender, Y - Joined MDT-data group, y - Sending to MDT-data group, G - Received BGP C-Mroute, g - Sent BGP C-Mroute, N - Received BGP Shared-Tree Prune, n - BGP C-Mroute suppressed, Q - Received BGP S-A Route, q - Sent BGP S-A Route, V - RD & Vector, v - Vector, p - PIM Joins on route, x - VxLAN group, c - PFP-SA cache created entry Outgoing interface flags: H - Hardware switched, A - Assert winner, p - PIM Join Timers: Uptime/Expires Interface state: Interface, Next-Hop or VCD, State/Mode (10.255.1.1, 239.10.10.10), 00:00:25/00:02:34, flags: FTx <<— x flag set for VxLAN group Incoming interface: Loopback999, RPF nbr 0.0.0.0 <<— Broadcast being encapsulated into VXLAN tunnel IP Outgoing interface list: TwoGigabitEthernet1/0/2, Forward/Sparse, 00:00:23/00:03:06 <<— Sending towards core to VTEP-2 (10.255.1.4, 239.10.10.10), 3d18h/00:02:25, flags: JTx <<— BUM traffic from VTEP-2 (if the ARP request was from VTEP-2) Incoming interface: TwoGigabitEthernet1/0/2, RPF nbr 10.1.1.6 Outgoing interface list: Tunnel0, Forward/Sparse-Dense, 3d18h/00:00:14 <<— Tunnel 0 is the VXLAN tunnel used for decapsulation
Källa: Cisco troubleshooting_bgp_evpn_vxlan
VXLAN tunnels are established using BGP EVPN, leaf nodes can dynamically learn the MAC addresses of hosts through BGP EVPN routes. The whole original IP-packet is encapsulated.
Källa: Huawei understanding-vxlan-tunnel-establishment-in-bgp-evpn-mode-distributed-vxlan-gateway
VXLAN (Virtual Extensible LAN) - The technology that provides the same Ethernet Layer 2 network services as VLAN does today, but with greater extensibility and flexibility.
VTEP (Virtual Tunnel Endpoint) - This is the device that does the encapsulation and de-encapsulation.
NVE (Network Virtual Interface) - Logical interface where the encapsulation and de-encapsulation occur.
EVI (EVPN instance): A logical switch within the EVPN domain which spans and interconnects multiple VTEPs to provide tenant Layer 2 and Layer 3 connectivity.
VNI (Vxlan Network Identifier) - 24 bit segment ID that defines the broadcast domain. Interchangeable with "VXLAN Segment ID".
Completely different 🫔 enchilada!
Next lab
Tomorrow you can do a completely different VXLAN lab with L3, VRF, Anycast Gateway!
Have a go at csepracticals.com bgp evpn vxlan lab whis is also in 3 parts: "Part 1 - Intra Vlan Routing", "Part 2 - BGP-EVPN Control Plane & Route Advertisement" and "Part 3 - Inter VNI Routing"
(BACKUP: vxlan-part1.pdf vxlan-part2.pdf vxlan-part3.pdf )
Thank you for your interest in EVPN and VXLANs !
• Part1: Underlay --> Part2: Tunnel --> You Are Here == Finished!
OBS chatten skiljer inte på loopback1 och loopback1 som skall vara olika !!
This conversation was generated with Duck.ai (https://duck.ai) using OpenAI's GPT-4o mini Model. AI chats may display inaccurate or offensive information (see https://duckduckgo.com/duckai/privacy-terms for more info).
====================
User prompt 1 of 10 - 2025-10-02, 09:58:32:
on a cisco 3650 when configuring mp bgp evpn i get the message "Can't configure L2DP VNIs on NVE interface with CP or CP VNI configured". This happened after upgrade of IOS
GPT-4o mini:
The error message "Can't configure L2DP VNIs on NVE interface with CP or CP VNI configured" indicates that there is a conflict in your configuration related to the Virtual Network Identifier (VNI) settings on the NVE (Network Virtualization Edge) interface after your IOS upgrade. This typically occurs when the configuration is not compatible with the new IOS version or when certain features are not supported together.
show running-config interface nve1
no vni <vni-number>
Here’s a simplified example of how you might configure an NVE interface for EVPN without conflicts:
interface nve1
no shutdown
host-reachability protocol bgp
member vni <your-vni>
l2vpn evpn
If you continue to experience issues after following these steps, consider reaching out to Cisco support for further assistance, as they can provide more tailored guidance based on your specific setup and IOS version.
User prompt 2 of 10 - 2025-10-02, 09:59:07:
what is CP
GPT-4o mini:
In the context of Cisco's EVPN (Ethernet Virtual Private Network) and NVE (Network Virtualization Edge) configurations, CP stands for Control Plane.
Functionality: The Control Plane is responsible for managing the signaling and control messages that establish and maintain the network's virtualized environment. It handles tasks such as route distribution, neighbor discovery, and the establishment of virtual network identifiers (VNIs).
Role in EVPN: In EVPN, the Control Plane is used to distribute MAC address information and other necessary data across the network. This allows for efficient Layer 2 and Layer 3 services over a shared infrastructure.
Configuration: When configuring EVPN, you can specify VNIs for the Control Plane, which are used to identify specific virtual networks. This is essential for ensuring that traffic is correctly routed and managed within the virtualized environment.
Separation from Data Plane: The Control Plane operates separately from the Data Plane, which is responsible for the actual forwarding of packets. This separation allows for more efficient management and scalability of the network.
Conflict with L2DP: As indicated by the error message you encountered, configuring both Control Plane and L2 Data Plane (L2DP) VNIs on the same NVE interface can lead to conflicts, as they serve different purposes and cannot coexist in that context.
Understanding the role of the Control Plane is crucial for effectively configuring and troubleshooting EVPN and NVE setups in Cisco environments.
User prompt 3 of 10 - 2025-10-02, 10:00:35:
how do i turn off CP
GPT-4o mini:
To turn off the Control Plane (CP) functionality on a Cisco device when configuring EVPN (Ethernet Virtual Private Network) with NVE (Network Virtualization Edge), you typically need to remove or disable the specific configuration related to the Control Plane VNI. Here’s how you can do that:
configure terminal
nve1
, use the following command:interface nve1
no vni <vni-number>
<vni-number>
with the actual VNI number you want to remove.exit
show running-config interface nve1
write memory
Here’s an example of how the commands might look:
configure terminal
interface nve1
no vni 100
exit
write memory
By following these steps, you should be able to successfully turn off the Control Plane functionality on your Cisco device.
User prompt 4 of 10 - 2025-10-02, 10:01:28:
what is l2dp
GPT-4o mini:
L2DP stands for Layer 2 Data Plane. In the context of networking, particularly with Cisco's EVPN (Ethernet Virtual Private Network) and NVE (Network Virtualization Edge) configurations, L2DP refers to the mechanisms and protocols used to handle the forwarding of Layer 2 traffic across a network.
Data Forwarding: L2DP is responsible for the actual forwarding of Ethernet frames between devices in a Layer 2 network. It deals with the transmission of data packets based on MAC addresses.
Separation from Control Plane: L2DP operates independently of the Control Plane (CP). While the Control Plane manages signaling, routing, and control messages, the Data Plane focuses solely on the efficient delivery of user data.
Integration with EVPN: In an EVPN setup, L2DP is used to transport Layer 2 traffic over an IP network. EVPN leverages L2DP to provide services such as MAC address learning, broadcast, and multicast handling.
VNI Configuration: In the context of EVPN, you can configure Layer 2 Data Plane VNIs (Virtual Network Identifiers) to identify specific Layer 2 segments. This allows for the segmentation of traffic and the creation of virtualized Layer 2 networks over a shared infrastructure.
Control Plane (CP): Manages the signaling and control messages necessary for establishing and maintaining the network. It handles tasks like route distribution and neighbor discovery.
Layer 2 Data Plane (L2DP): Focuses on the actual forwarding of data packets. It is responsible for the transmission of user data across the network.
In a typical EVPN deployment, you might configure both Control Plane and Data Plane VNIs. For instance, you could have a Control Plane VNI for signaling and a separate Data Plane VNI for forwarding traffic. However, as indicated by the error message you encountered, you cannot configure both types of VNIs on the same NVE interface simultaneously.
L2DP is a crucial component of modern networking, especially in virtualized environments where Layer 2 services are required over Layer 3 infrastructures. Understanding L2DP and its role in conjunction with the Control Plane is essential for effective network design and troubleshooting in Cisco environments.
User prompt 5 of 10 - 2025-10-02, 10:02:46:
can i have the same loopback address for both CP and L2DP
GPT-4o mini:
No, you cannot use the same loopback address for both Control Plane (CP) and Layer 2 Data Plane (L2DP) in a Cisco EVPN (Ethernet Virtual Private Network) configuration. Each function typically requires its own unique loopback address for proper operation and to avoid conflicts.
When configuring your EVPN setup, you should assign different loopback addresses for CP and L2DP. Here’s an example:
interface Loopback0
ip address 10.1.1.1 255.255.255.255 ! Control Plane Loopback
interface Loopback1
ip address 10.1.1.2 255.255.255.255 ! Data Plane Loopback
To ensure proper functionality and avoid conflicts, always use unique loopback addresses for Control Plane and Layer 2 Data Plane configurations in your Cisco EVPN setup. This practice will help maintain a stable and efficient network environment.
User prompt 6 of 10 - 2025-10-02, 10:03:28:
what is the complete CP configuration on a 3650
GPT-4o mini:
To configure the Control Plane (CP) for EVPN (Ethernet Virtual Private Network) on a Cisco 3650 switch, you need to set up the NVE (Network Virtualization Edge) interface, define the loopback interface for the Control Plane, and configure the necessary parameters for EVPN. Below is a complete example of a typical CP configuration.
First, create a loopback interface that will be used for the Control Plane.
interface Loopback0
ip address 10.1.1.1 255.255.255.255 ! Control Plane Loopback
Next, configure the NVE interface to enable EVPN and associate it with the loopback interface.
interface nve1
no shutdown
host-reachability protocol bgp
source-interface Loopback0
Define the EVPN instance and specify the Virtual Network Identifier (VNI) for the Control Plane.
evpn
vni 1000 l2
rd 10.1.1.1:1000 ! Route Distinguisher
route-target both 1000 ! Route Target
Set up BGP to advertise the EVPN routes. This includes defining the BGP router ID and the EVPN address family.
router bgp 65000
bgp log-neighbor-changes
neighbor 10.1.1.2 remote-as 65001 ! Neighbor configuration
address-family l2vpn evpn
neighbor 10.1.1.2 activate
advertise-all-vni
exit-address-family
You can also configure the VNI for the Control Plane if needed.
interface nve1
member vni 1000
l2vpn evpn
After completing the configuration, verify it using the following commands:
show running-config
show ip bgp summary
show evpn vni
This configuration provides a basic setup for the Control Plane in an EVPN environment on a Cisco 3650 switch. Adjust the IP addresses, AS numbers, and VNIs according to your specific network design and requirements.
User prompt 7 of 10 - 2025-10-02, 10:04:31:
what is the complete l2DP configuration, that does not collide with CP config
GPT-4o mini:
To configure the Layer 2 Data Plane (L2DP) for EVPN (Ethernet Virtual Private Network) on a Cisco 3650 switch without colliding with the Control Plane (CP) configuration, you need to ensure that the configurations for both planes are distinct and do not share the same parameters, such as loopback addresses or VNIs. Below is a complete example of a typical L2DP configuration.
Create a separate loopback interface that will be used for the Data Plane.
interface Loopback1
ip address 10.1.2.1 255.255.255.255 ! Data Plane Loopback
Next, configure the NVE interface to enable EVPN and associate it with the loopback interface for the Data Plane.
interface nve1
no shutdown
host-reachability protocol bgp
source-interface Loopback1
Define the EVPN instance and specify the Virtual Network Identifier (VNI) for the Data Plane.
evpn
vni 2000 l2
rd 10.1.2.1:2000 ! Route Distinguisher for L2DP
route-target both 2000 ! Route Target for L2DP
Set up BGP to advertise the EVPN routes for the Data Plane. This includes defining the BGP router ID and the EVPN address family.
router bgp 65000
bgp log-neighbor-changes
neighbor 10.1.2.2 remote-as 65001 ! Neighbor configuration for L2DP
address-family l2vpn evpn
neighbor 10.1.2.2 activate
advertise-all-vni
exit-address-family
You can also configure the VNI for the Data Plane.
interface nve1
member vni 2000
l2vpn evpn
After completing the configuration, verify it using the following commands:
show running-config
show ip bgp summary
show evpn vni
This configuration provides a basic setup for the Layer 2 Data Plane in an EVPN environment on a Cisco 3650 switch. Adjust the IP addresses, AS numbers, and VNIs according to your specific network design and requirements, ensuring that they do not overlap with the Control Plane configuration.
User prompt 8 of 10 - 2025-10-02, 10:13:34:
which loopback should have "ip pim sparse mode" CP or L2DP
GPT-4o mini:
The loopback interface that should have the command ip pim sparse-mode
is typically the Control Plane (CP) loopback interface. This is because the Control Plane is responsible for managing multicast routing and signaling, which is essential for protocols like PIM (Protocol Independent Multicast).
Here’s how you might configure the Control Plane loopback with PIM sparse mode:
interface Loopback0
ip address 10.1.1.1 255.255.255.255 ! Control Plane Loopback
ip pim sparse-mode
In summary, you should configure ip pim sparse-mode
on the Control Plane (CP) loopback interface. This setup ensures that multicast routing is effectively managed and that the Control Plane can handle the necessary signaling for multicast traffic. The Data Plane loopback should not have PIM configured, as it is focused on data forwarding rather than control and signaling functions.
User prompt 9 of 10 - 2025-10-02, 10:51:01:
is there any other root cause then use of loopbacks to this problem
GPT-4o mini:
Yes, there are several potential root causes for issues related to the configuration of Control Plane (CP) and Layer 2 Data Plane (L2DP) in an EVPN (Ethernet Virtual Private Network) setup on a Cisco 3650 switch, beyond just the use of loopback interfaces. Here are some other common factors that could lead to problems:
While loopback configuration is a critical aspect of setting up CP and L2DP, other factors such as VNI conflicts, BGP misconfigurations, routing protocol issues, and resource limitations can also contribute to problems in an EVPN setup. A thorough review of the entire configuration and network environment is essential to identify and resolve these issues effectively.
User prompt 10 of 10 - 2025-10-02, 10:55:02:
what is the complete l2DP configuration, that does not collide with CP config, but make sure it is catalyst config not nexus
GPT-4o mini:
To configure the Layer 2 Data Plane (L2DP) for EVPN (Ethernet Virtual Private Network) on a Cisco Catalyst 3650 switch, ensuring that it does not collide with the Control Plane (CP) configuration, you need to follow a structured approach. Below is a complete example of a typical L2DP configuration specifically for a Catalyst environment.
Create a separate loopback interface that will be used for the Data Plane. This loopback should be distinct from the Control Plane loopback.
interface Loopback1
ip address 10.1.2.1 255.255.255.255 ! Data Plane Loopback
Next, configure the NVE interface to enable EVPN and associate it with the loopback interface for the Data Plane.
interface nve1
no shutdown
host-reachability protocol bgp
source-interface Loopback1
Define the EVPN instance and specify the Virtual Network Identifier (VNI) for the Data Plane. Ensure that the VNI is unique and does not overlap with the Control Plane VNI.
evpn
vni 2000 l2
rd 10.1.2.1:2000 ! Route Distinguisher for L2DP
route-target both 2000 ! Route Target for L2DP
Set up BGP to advertise the EVPN routes for the Data Plane. This includes defining the BGP router ID and the EVPN address family.
router bgp 65000
bgp log-neighbor-changes
neighbor 10.1.2.2 remote-as 65001 ! Neighbor configuration for L2DP
address-family l2vpn evpn
neighbor 10.1.2.2 activate
advertise-all-vni
exit-address-family
You can also configure the VNI for the Data Plane within the NVE interface.
interface nve1
member vni 2000
l2vpn evpn
After completing the configuration, verify it using the following commands:
show running-config
show ip bgp summary
show evpn vni
This configuration provides a basic setup for the Layer 2 Data Plane in an EVPN environment on a Cisco Catalyst 3650 switch. Adjust the IP addresses, AS numbers, and VNIs according to your specific network design and requirements, ensuring that they do not overlap with the Control Plane configuration.
• Part1: Underlay --> Part2: Tunnel --> You Are Here == Finished!