From the NSX 3.0 version, we can configure the EVPN function between the T0 router and the physical router. This post will be devoted to how to configure EVPN step by step in inline mode.
What is EVPN?
EVPN (Ethernet VPN) is an IEEE standard that allows for the creation of virtual private networks (VPNs) at the Layer 2 (LAN) and Layer 3 (IP) levels for Ethernet devices. EVPN uses two planes: the control plane and the data plane. The control plane is based on the Multiprotocol Border Gateway Protocol (MP-BGP), which is responsible for distributing MAC and IP address information between devices in the network. The data plane uses Virtual Extensible LAN (VXLAN) technology, which allows Ethernet packets to be transmitted within a tunnel network.
Inline Mode
In this mode, the tier-0 gateway establishes MP-BGP EVPN control plane sessions with external routers to exchange routing information. In the data plane, edge nodes forward all traffic exiting the local data center to the data center gateways and incoming traffic from the remote data center to the hypervisors in the local data center. As the edge nodes are in the data forwarding path, this model is called the Inline model.
Test architecture NSX-T EVPN
In the solution architecture, two Virtual Routing and Forwarding (VRF) instances are defined on the external router (e.g., Juniper vMX), which are then passed to the T0 router through an MP-BGP session. On the T0 router, these VRF instances are extended to corresponding VRFs with the same names and Route Distinguisher (RD) identifiers. Then, the T1 routers are connected to these VRFs, which will have connected network segments. As a result, each network segment can function as an independent VRF, and devices in different segments can communicate with each other as if they were in a single physical LAN.
Limitations and Caveats
According to VMware documentation.
| Limitations | Caveats |
|
|
NSX configuration
VNI POOL configuration
We start our work by logging into NSX Magera. Then go to Networking -> Networking Settings. Click VNI Pool.
We will create a VNI pool that will be used in EVPN. Locate the option to create a new VNI pool and click on “Add VNI POOL” to begin defining your VNI ID pool.
We provide:
Pool name
Start VNI ID
End of VNI ID
Segment configuration
The next element is to configure the vlan segment, it will be used to create an external interface in T0.
We go to “Networking” then “Segments”
Click ADD Segment in the new window to configure it.
We provide:
Segment name
select Transport Zone – our TZ type vlan
VLAN ID
Create and configure the T0 router
Within this environment, we have one edge VM assigned to the cluster – edge-cluster-02. Router T0 will be created on this cluster. Below are the steps to create and configure a T0 router from EVPN to vMX Router.
- Create Router T0
We go to “Networking” then “Tier-0 Gateways” click “ADD Gateway”, select “Tier-0”
- In the new window, enter the parameters for T0
We enter the parameters:
Name T0
mode
Edge Cluster
After entering the data, click save.
Now we move on to the configuration of the T0 router.
- Create an external interface
Go to the T0 edition, then go to the Interfaces section and click “Set”
- In the new window, find “ADD Interface” then configure our external interface.
We provide configuration data:
– segment name
– type in this case external
– IP address
– we assign to the segment created earlier.
– EDGE Node
– MTU - The next interface to create will be loopback. It will help us to compile MPBGP sessions with EVPN. Steps as above but choose the loopback type.
- EVPN configuration at T0 level.
In the T0 edition, go to the “EVPN Settings” section. Where we configure:
VNI Pool – select the pool that we created earlier.
EVPN Endpoint Tunnel - EVPN Endpoint Tunnel
Here we configure the EVPN Tunnel Endpoint, it is the same as the loopback we created earlier. - BGP
We start the BGP configuration by assigning the ASN number. Then we move on to BGP Neighbors. - The first session will be between T0 and vMX on the uplink.
Neighborhood configuration
We provide the data:
Peer’s IP
Neighbor’s ASN
source IP - Second BGP session between T0 and vMX using loopback as peer addresses.
We provide the data:
Peer’s IP
Neighbor’s ASN
source IP
We add a filter L2VPN_EVPN for this session:
- Finally, we configure the redistribution of prefixes.
In the Route RE-DISTRBUTION section, we configure the profile where we enable the following redistributions:
Thanks to the steps above, we have an MP-BGP configuration between T0 and vMX.
According to the drawing, we need to prepare two VRFs A and B. Below is their configuration.
VRF A
- Create VRF A
We go to “Networking” then “Tier-0 Gateways” click “ADD Gateway”, select “VRF”
In the new window, we configure the name for the VRF to which T0 will be connected, We assign a unique RD (Route Distinguisher) and select the VNI ID that will be used for tunneling between the VRF and vMX
Here you can apply a method for an RD that is in AS:VNI format.
- I’m creating a loopback interface for testing
- We enable the redistribution of loopback connected networks and networks with T1 at the VRF level.
The configuration is shown for VRF A. The steps for VRF B are the same, only we change the data as shown in the picture below.
T1 router
According to the plan, a Tier 1 router has been connected to each VRF. Below is the configuration for VRF A.
- Create Tier 1
We go to “Networking” then “Tier-1 Gateways” click “ADD TIER-1 GATEWAY”
- We configure Route advertising
- We create a segment connected to T1
The above steps are repeated for Router T1 connected to VRF B.
vMX configuration
- Interface configuration ge-0/0/0.11 to T0.
We configure the connection between T0 and vMX and enable MTU 9K on the interface.set interfaces ge-0/0/0 vlan-tagging set interfaces ge-0/0/0 mtu 9000 set interfaces ge-0/0/0 unit 11 vlan-id 11 set interfaces ge-0/0/0 unit 11 family inet address 192.168.10.1/24
- Configuration of the loopback.0 interface
set interfaces lo0 unit 0 family inet address 10.11.10.2/32
- Loopback configuration for VRF A and B
set interfaces lo0 unit 100 family inet address 172.30.100.2/32 set interfaces lo0 unit 101 family inet address 192.168.100.2/32
- Configuring a BGP session on the ge-0/0/0.11 connection
set protocols bgp group T0 type internal set protocols bgp group T0 family inet unicast set protocols bgp group T0 peer-as 65000 set protocols bgp group T0 neighbor 192.168.10.10
- BGP session setup on loopback.0 connection with epn signaling
set protocols bgp group EVPN type internal set protocols bgp group EVPN family inet unicast set protocols bgp group EVPN family evpn signaling set protocols bgp group EVPN neighbor 10.11.10.1 local-address 10.11.10.2 set protocols bgp group EVPN neighbor 10.11.10.1 peer-as 65000
- Configuration of the export policy of all direct networks. It will be used below when configuring the VRF
set policy-options policy-statement export_direct from protocol direct set policy-options policy-statement export_direct then accept
- VRF Configuration – VRF A
set routing-instances vrfx-a protocols evpn ip-prefix-routes advertise direct-nexthop set routing-instances vrfx-a protocols evpn ip-prefix-routes encapsulation vxlan set routing-instances vrfx-a protocols evpn ip-prefix-routes vni 95001 set routing-instances vrfx-a protocols evpn ip-prefix-routes export export_direct set routing-instances vrfx-a vtep-source-interface lo0.0 set routing-instances vrfx-a instance-type vrf set routing-instances vrfx-a interface lo0.100 set routing-instances vrfx-a route-distinguisher 65000:95001 set routing-instances vrfx-a vrf-target target:65000:95001
- VRF Configuration – VRF B
set routing-instances vrfx-b protocols evpn ip-prefix-routes advertise direct-nexthop set routing-instances vrfx-b protocols evpn ip-prefix-routes encapsulation vxlan set routing-instances vrfx-b protocols evpn ip-prefix-routes vni 95002 set routing-instances vrfx-b protocols evpn ip-prefix-routes export export_direct set routing-instances vrfx-b vtep-source-interface lo0.0 set routing-instances vrfx-b instance-type vrf set routing-instances vrfx-b interface lo0.101 set routing-instances vrfx-b route-distinguisher 65000:95002 set routing-instances vrfx-b vrf-target target:65000:95002
- We make a commit and proceed to verification.
Verification
Verification on the vMX side
- BGP
root> show bgp summary Threading mode: BGP I/O Groups: 2 Peers: 2 Down peers: 0 Table Tot Paths Act Paths Suppressed History Damp State Pending inet.0 2 0 0 0 0 0 bgp.evpn.0 4 4 0 0 0 0 Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped... 10.11.10.1 65000 30 39 0 2 10:38 Establ inet.0: 0/1/1/0 bgp.evpn.0: 4/4/4/0 vrfx-a.evpn.0: 2/2/2/0 vrfx-b.evpn.0: 2/2/2/0 192.168.10.10 65000 24 27 0 2 10:38 Establ inet.0: 0/1/1/0As we can see BGP unicast and evpn sessions are set up.
- We check the routing table
root> show route | no-more inet.0: 14 destinations, 16 routes (14 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 0.0.0.0/0 *[Static/5] 5d 05:20:34 > to 10.101.9.1 via fxp0.0 10.11.10.1/32 *[Static/5] 3d 00:51:26 > to 192.168.10.10 via ge-0/0/0.11 [BGP/170] 00:11:42, MED 0, localpref 100, from 10.11.10.1 AS path: ?, validation-state: unverified > to 192.168.10.10 via ge-0/0/0.11 [BGP/170] 00:11:42, MED 0, localpref 100 AS path: ?, validation-state: unverified > to 192.168.10.10 via ge-0/0/0.11 vrfx-a.inet.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 172.30.30.0/24 *[EVPN/170] 00:09:56 > to 192.168.10.10 via ge-0/0/0.11 172.30.100.1/32 *[EVPN/170] 00:09:56 > to 192.168.10.10 via ge-0/0/0.11 172.30.100.2/32 *[Direct/0] 00:14:31 > via lo0.100 vrfx-b.inet.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 192.168.2.0/24 *[EVPN/170] 00:08:41 > to 192.168.10.10 via ge-0/0/0.11 192.168.100.1/32 *[EVPN/170] 00:08:41 > to 192.168.10.10 via ge-0/0/0.11 192.168.100.2/32 *[Direct/0] 00:14:31 > via lo0.101 ..... bgp.evpn.0: 6 destinations, 6 routes (6 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 5:65000:95001::0::172.30.30.0::24/248 *[BGP/170] 00:33:41, MED 0, localpref 100, from 10.11.10.1 AS path: ?, validation-state: unverified > to 192.168.10.10 via ge-0/0/0.11 5:65000:95001::0::172.30.100.1::32/248 *[BGP/170] 00:33:41, MED 0, localpref 100, from 10.11.10.1 AS path: ?, validation-state: unverified > to 192.168.10.10 via ge-0/0/0.11 5:65000:95001::0::172.30.100.2::32/248 *[EVPN/170] 00:03:41 Indirect 5:65000:95002::0::192.168.2.0::24/248 *[BGP/170] 00:32:26, MED 0, localpref 100, from 10.11.10.1 AS path: ?, validation-state: unverified > to 192.168.10.10 via ge-0/0/0.11 5:65000:95002::0::192.168.100.1::32/248 *[BGP/170] 00:32:26, MED 0, localpref 100, from 10.11.10.1 AS path: ?, validation-state: unverified > to 192.168.10.10 via ge-0/0/0.11 5:65000:95002::0::192.168.100.2::32/248 *[EVPN/170] 00:03:07 Indirect vrfx-a.evpn.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 5:65000:95001::0::172.30.30.0::24/248 *[BGP/170] 00:00:05, MED 0, localpref 100, from 10.11.10.1 AS path: ?, validation-state: unverified > to 192.168.10.10 via ge-0/0/0.11 5:65000:95001::0::172.30.100.1::32/248 *[BGP/170] 00:00:05, MED 0, localpref 100, from 10.11.10.1 AS path: ?, validation-state: unverified > to 192.168.10.10 via ge-0/0/0.11 5:65000:95001::0::172.30.100.2::32/248 *[EVPN/170] 00:03:41 Indirect vrfx-b.evpn.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 5:65000:95002::0::192.168.2.0::24/248 *[BGP/170] 00:00:05, MED 0, localpref 100, from 10.11.10.1 AS path: ?, validation-state: unverified > to 192.168.10.10 via ge-0/0/0.11 5:65000:95002::0::192.168.100.1::32/248 *[BGP/170] 00:00:05, MED 0, localpref 100, from 10.11.10.1 AS path: ?, validation-state: unverified > to 192.168.10.10 via ge-0/0/0.11 5:65000:95002::0::192.168.100.2::32/248 *[EVPN/170] 00:03:07 IndirectEverything looks ok in the routing table – as you can see only in the inet.0 table, it is worth turning on the appropriate filtering of broadcast prefixes in BGP for the loopback from the T0 side, but this is cosmetic.
- favorite ping test between loopback interfaces
- VRF A
root> ping 192.168.100.1 routing-instance vrfx-b PING 192.168.100.1 (192.168.100.1): 56 data bytes 64 bytes from 192.168.100.1: icmp_seq=0 ttl=64 time=1.266 ms 64 bytes from 192.168.100.1: icmp_seq=1 ttl=64 time=1.429 ms 64 bytes from 192.168.100.1: icmp_seq=2 ttl=64 time=1.347 ms 64 bytes from 192.168.100.1: icmp_seq=3 ttl=64 time=1.386 ms 64 bytes from 192.168.100.1: icmp_seq=4 ttl=64 time=1.698 ms 64 bytes from 192.168.100.1: icmp_seq=5 ttl=64 time=1.530 ms 64 bytes from 192.168.100.1: icmp_seq=6 ttl=64 time=1.450 ms ^C --- 192.168.100.1 ping statistics --- 7 packets transmitted, 7 packets received, 0% packet loss round-trip min/avg/max/stddev = 1.266/1.444/1.698/0.129 ms
- VRF B
root> ping 172.30.100.1 routing-instance vrfx-a PING 172.30.100.1 (172.30.100.1): 56 data bytes 64 bytes from 172.30.100.1: icmp_seq=0 ttl=64 time=1.325 ms 64 bytes from 172.30.100.1: icmp_seq=1 ttl=64 time=1.249 ms 64 bytes from 172.30.100.1: icmp_seq=2 ttl=64 time=1.390 ms ^C --- 172.30.100.1 ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max/stddev = 1.249/1.321/1.390/0.058 ms
- VRF A
NSX-T side verification
On the NSX side, we check from the CLI. We log in from the EDGE VM on which T0 is running.
- We check which id is running SR for T0
dc1edge03> get logical-router Logical Router UUID VRF LR-ID Name Type Ports 736a80e3-23f6-5a2d-81d6-bbefb2786666 0 0 TUNNEL 4 f9414cef-e3e3-405a-a7f8-55f349e08f22 1 1026 SR-VRF-VRF-A VRF_SERVICE_ROUTER_TIER0 6 11ab6c6f-222b-434f-b25b-aad51dadf2ed 2 1027 SR-VRF-VRF-B VRF_SERVICE_ROUTER_TIER0 6 987450ee-042c-41aa-9d0d-d4fcabd44b36 3 25 SR-R-T1-VRF-A SERVICE_ROUTER_TIER1 5 06d10402-2066-4e4f-8243-6679d1bcb6b3 4 14 SR-R-T0-MPBGP SERVICE_ROUTER_TIER0 7 f2ca25ae-49a3-4b12-a91e-c5c9908bb6c9 6 22 DR-VRF-VRF-A VRF_DISTRIBUTED_ROUTER_TIER0 4 c43a9c6d-b207-4022-b43c-e52728eb1ab5 7 24 DR-R-T1-VRF-A DISTRIBUTED_ROUTER_TIER1 4 304d5182-a40c-486a-a1e3-6d35275951fd 8 27 SR-R-T1-VRF-B SERVICE_ROUTER_TIER1 5 758b42b3-8a4c-452f-91ad-6baf97188879 9 23 DR-VRF-VRF-B VRF_DISTRIBUTED_ROUTER_TIER0 4 6204d89c-ec0e-40a9-9824-33adbbb41bff 10 26 DR-R-T1-VRF-B DISTRIBUTED_ROUTER_TIER1 4
- go to VRF 4 (SR T0) and check BGP status
dc1edge03(tier0_sr)> get bgp neighbor summary BFD States: NC - Not configured, AC - Activating,DC - Disconnected AD - Admin down, DW - Down, IN - Init,UP - Up BGP summary information for VRF default for address-family: ipv4Unicast Router ID: 192.168.10.10 Local AS: 65000 Neighbor AS State Up/DownTime BFD InMsgs OutMsgs InPfx OutPfx 10.11.10.2 65000 Estab 00:20:34 NC 9887 8712 0 0 192.168.10.1 65000 Estab 00:20:34 NC 9869 8910 1 0 BFD States: NC - Not configured, AC - Activating,DC - Disconnected AD - Admin down, DW - Down, IN - Init,UP - Up BGP summary information for VRF default for address-family: l2VpnEvpn Router ID: 192.168.10.10 Local AS: 65000 Neighbor AS State Up/DownTime BFD InMsgs OutMsgs InPfx OutPfx 10.11.10.2 65000 Estab 00:20:34 NC 9887 8712 0 0 192.168.10.1 65000 Estab 00:20:34 NC 9869 8910 0 0 - check status of BGP EVPN – get bgp evpn
dc1edge03(tier0_sr)> get bgp evpn BGP table version is 3, local router ID is 192.168.10.10 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal Origin codes: i - IGP, e - EGP, ? - incomplete EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP] EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP] EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP] EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP] Network Next Hop Metric LocPrf Weight Path Route Distinguisher: 65000:95001 *> [5]:[0]:[24]:[172.30.30.0] 10.11.10.1 0 32768 ? *> [5]:[0]:[32]:[172.30.100.1] 10.11.10.1 0 32768 ? *>i[5]:[0]:[32]:[172.30.100.2] 10.11.10.2 100 0 i Route Distinguisher: 65000:95002 *> [5]:[0]:[24]:[192.168.2.0] 10.11.10.1 0 32768 ? *> [5]:[0]:[32]:[192.168.100.1] 10.11.10.1 0 32768 ? *>i[5]:[0]:[32]:[192.168.100.2] 10.11.10.2 100 0 i Displayed 6 prefixes (6 paths)Some other helpful commands to verify from the T0 router (SR)
Displays all available VRFs with assigned VNI get bgp evpn vni Displays the BGP routing table by RD get bgp evpn overlay Shows configured VRFs that are active in EVPN get evpn vni
