nsx-t logo

NSX-T EVPN Inline Mode

From the NSX 3.0 version, we can configure the EVPN function between the T0 router and the physical router. This post will be devoted to how to configure EVPN step by step in inline mode.

What is EVPN?

EVPN (Ethernet VPN) is an IEEE standard that allows for the creation of virtual private networks (VPNs) at the Layer 2 (LAN) and Layer 3 (IP) levels for Ethernet devices. EVPN uses two planes: the control plane and the data plane. The control plane is based on the Multiprotocol Border Gateway Protocol (MP-BGP), which is responsible for distributing MAC and IP address information between devices in the network. The data plane uses Virtual Extensible LAN (VXLAN) technology, which allows Ethernet packets to be transmitted within a tunnel network.

Inline Mode

In this mode, the tier-0 gateway establishes MP-BGP EVPN control plane sessions with external routers to exchange routing information. In the data plane, edge nodes forward all traffic exiting the local data center to the data center gateways and incoming traffic from the remote data center to the hypervisors in the local data center. As the edge nodes are in the data forwarding path, this model is called the Inline model.

Test architecture NSX-T EVPN

In the solution architecture, two Virtual Routing and Forwarding (VRF) instances are defined on the external router (e.g., Juniper vMX), which are then passed to the T0 router through an MP-BGP session. On the T0 router, these VRF instances are extended to corresponding VRFs with the same names and Route Distinguisher (RD) identifiers. Then, the T1 routers are connected to these VRFs, which will have connected network segments. As a result, each network segment can function as an independent VRF, and devices in different segments can communicate with each other as if they were in a single physical LAN.

 

Limitations and Caveats

According to VMware documentation. 

Limitations Caveats
  • NSX supports L3 VPN by advertising and receiving IP prefixes as EVPN Route Type-5.
  • NSX-T generates a unique route MAC for every NSX Edge VTEP in the EVPN domain. However, there may be other nodes in the network that are not managed by NSX-T, for example, physical routers. You must make sure that the router MACs are unique across all the VTEPs in the EVPN domain.
  • The EVPN feature supports NSX Edge nodes to be either the ingress or the egress of the EVPN virtual tunnel endpoint. If an NSX Edge node receives EVPN Route Type-5 prefixes from its eBGP peer that needs to be redistributed to another eBGP peer, the routes are re-advertised without any change to the next hop.
  • In multi-path network topologies, it is recommended that ECMP is enabled for the NSX BGP EVPN control plane, so that all the possible paths can be advertised by the tier-0 gateway. This will avoid any potential traffic blackhole due to asymmetric data path forwarding.
  • A tier-0 gateway can span across multiple edge nodes. However, specifying a unique route distinguisher for each edge node or TEP (either via auto or manual configuration) is not supported. As a result, the use of ECMP on the peer router is not supported.
  • Route maps are not supported for EVPN address family.
  • Recursive route resolution for gateway IP via default static route is not supported.
  • Only BGP Graceful Restart in Helper Mode is supported.
  • Only eBGP is supported between tier-0 SRs and external routers.
  • Only one TEP is supported per edge node. The use of loopback interfaces for TEP is highly recommended.

 

NSX configuration

VNI POOL configuration

We start our work by logging into NSX Magera. Then go to Networking -> Networking Settings. Click VNI Pool.


We will create a VNI pool that will be used in EVPN. Locate the option to create a new VNI pool and click on “Add VNI POOL”  to begin defining your VNI ID pool.

We provide:
Pool name
Start VNI ID
End of VNI ID

Segment configuration

The next element is to configure the vlan segment, it will be used to create an external interface in T0.

We go to “Networking” then “Segments”

Click ADD Segment in the new window to configure it.

We provide:
Segment name
select Transport Zone – our TZ type vlan
VLAN ID

 

Create and configure the T0 router

Within this environment, we have one edge VM assigned to the cluster – edge-cluster-02. Router T0 will be created on this cluster. Below are the steps to create and configure a T0 router from EVPN to vMX Router.

  1. Create Router T0
    We go to “Networking” then “Tier-0 Gateways” click  “ADD Gateway”, select “Tier-0”
  2. In the new window, enter the parameters for T0

    We enter the parameters:
    Name T0
    mode
    Edge Cluster
    After entering the data, click save.

Now we move on to the configuration of the T0 router.

  1. Create an external interface
    Go to the T0 edition, then go to the Interfaces section and click “Set”
  2. In the new window, find “ADD Interface” then configure our external interface.

    We provide configuration data:
    – segment name
    – type in this case external
    – IP address
    – we assign to the segment created earlier.
    – EDGE Node
    – MTU
  3. The next interface to create will be loopback. It will help us to compile MPBGP sessions with EVPN. Steps as above but choose the loopback type.
  4. EVPN configuration at T0 level.

    In the T0 edition, go to the “EVPN Settings” section. Where we configure:
    VNI Pool – select the pool that we created earlier.
    EVPN Endpoint Tunnel
  5. EVPN Endpoint Tunnel

    Here we configure the EVPN Tunnel Endpoint, it is the same as the loopback we created earlier.
  6. BGP
    We start the BGP configuration by assigning the ASN number. Then we move on to BGP Neighbors.
  7. The first session will be between T0 and vMX on the uplink.
    Schematic diagram for a bgp session
    Neighborhood configuration

    We provide the data:
    Peer’s IP
    Neighbor’s ASN
    source IP
  8. Second BGP session between T0 and vMX using loopback as peer addresses.


    We provide the data:
    Peer’s IP
    Neighbor’s ASN
    source IP
    We add a filter L2VPN_EVPN for this session:
  9. Finally, we configure the redistribution of prefixes.
    In the Route RE-DISTRBUTION section, we configure the profile where we enable the following redistributions:

Thanks to the steps above, we have an MP-BGP configuration between T0 and vMX.

According to the drawing, we need to prepare two VRFs A and B. Below is their configuration.

VRF A

  1. Create VRF A
    We go to “Networking” then “Tier-0 Gateways” click  “ADD Gateway”, select “VRF”

    In the new window, we configure the name for the VRF to which T0 will be connected, We assign a unique RD (Route Distinguisher) and select the VNI ID that will be used for tunneling between the VRF and vMX
    Here you can apply a method for an RD that is in AS:VNI format.
  2. I’m creating a loopback interface for testing
  3. We enable the redistribution of loopback connected networks and networks with T1 at the VRF level.

The configuration is shown for VRF A. The steps for VRF B are the same, only we change the data as shown in the picture below.

 

T1 router

According to the plan, a Tier 1 router has been connected to each VRF. Below is the configuration for VRF A.

  1. Create Tier 1
    We go to “Networking” then “Tier-1 Gateways” click “ADD TIER-1 GATEWAY”
  2. We configure Route advertising
  3. We create a segment connected to T1

The above steps are repeated for Router T1 connected to VRF B.

vMX configuration

  1. Interface configuration ge-0/0/0.11 to T0.
    We configure the connection between T0 and vMX and enable MTU 9K on the interface.

    set interfaces ge-0/0/0 vlan-tagging
    set interfaces ge-0/0/0 mtu 9000
    set interfaces ge-0/0/0 unit 11 vlan-id 11
    set interfaces ge-0/0/0 unit 11 family inet address 192.168.10.1/24

     

  2. Configuration of the loopback.0 interface
    set interfaces lo0 unit 0 family inet address 10.11.10.2/32
  3. Loopback configuration for VRF A and B
    set interfaces lo0 unit 100 family inet address 172.30.100.2/32
    set interfaces lo0 unit 101 family inet address 192.168.100.2/32
  4. Configuring a BGP session on the ge-0/0/0.11 connection
    set protocols bgp group T0 type internal
    set protocols bgp group T0 family inet unicast
    set protocols bgp group T0 peer-as 65000
    set protocols bgp group T0 neighbor 192.168.10.10
  5. BGP session setup on loopback.0 connection with epn signaling
    set protocols bgp group EVPN type internal
    set protocols bgp group EVPN family inet unicast
    set protocols bgp group EVPN family evpn signaling
    set protocols bgp group EVPN neighbor 10.11.10.1 local-address 10.11.10.2
    set protocols bgp group EVPN neighbor 10.11.10.1 peer-as 65000
  6. Configuration of the export policy of all direct networks. It will be used below when configuring the VRF
    set policy-options policy-statement export_direct from protocol direct
    set policy-options policy-statement export_direct then accept
  7. VRF Configuration – VRF A
    set routing-instances vrfx-a protocols evpn ip-prefix-routes advertise direct-nexthop
    set routing-instances vrfx-a protocols evpn ip-prefix-routes encapsulation vxlan
    set routing-instances vrfx-a protocols evpn ip-prefix-routes vni 95001
    set routing-instances vrfx-a protocols evpn ip-prefix-routes export export_direct
    set routing-instances vrfx-a vtep-source-interface lo0.0
    set routing-instances vrfx-a instance-type vrf
    set routing-instances vrfx-a interface lo0.100
    set routing-instances vrfx-a route-distinguisher 65000:95001
    set routing-instances vrfx-a vrf-target target:65000:95001
  8. VRF Configuration – VRF B
    set routing-instances vrfx-b protocols evpn ip-prefix-routes advertise direct-nexthop
    set routing-instances vrfx-b protocols evpn ip-prefix-routes encapsulation vxlan
    set routing-instances vrfx-b protocols evpn ip-prefix-routes vni 95002
    set routing-instances vrfx-b protocols evpn ip-prefix-routes export export_direct
    set routing-instances vrfx-b vtep-source-interface lo0.0
    set routing-instances vrfx-b instance-type vrf
    set routing-instances vrfx-b interface lo0.101
    set routing-instances vrfx-b route-distinguisher 65000:95002
    set routing-instances vrfx-b vrf-target target:65000:95002
  9. We make a commit and proceed to verification.

Verification

Verification on the vMX side

  1. BGP
    root> show bgp summary 
    Threading mode: BGP I/O
    Groups: 2 Peers: 2 Down peers: 0
    Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending
    inet.0               
                           2          0          0          0          0          0
    bgp.evpn.0           
                           4          4          0          0          0          0
    Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
    10.11.10.1            65000         30         39       0       2       10:38 Establ
      inet.0: 0/1/1/0
      bgp.evpn.0: 4/4/4/0
      vrfx-a.evpn.0: 2/2/2/0
      vrfx-b.evpn.0: 2/2/2/0
    192.168.10.10         65000         24         27       0       2       10:38 Establ
      inet.0: 0/1/1/0

    As we can see BGP unicast and evpn sessions are set up.

  2. We check the routing table
    root> show route | no-more 
    
    inet.0: 14 destinations, 16 routes (14 active, 0 holddown, 0 hidden)
    + = Active Route, - = Last Active, * = Both
    
    0.0.0.0/0          *[Static/5] 5d 05:20:34
                        >  to 10.101.9.1 via fxp0.0
    10.11.10.1/32      *[Static/5] 3d 00:51:26
                        >  to 192.168.10.10 via ge-0/0/0.11
                        [BGP/170] 00:11:42, MED 0, localpref 100, from 10.11.10.1
                          AS path: ?, validation-state: unverified
                        >  to 192.168.10.10 via ge-0/0/0.11
                        [BGP/170] 00:11:42, MED 0, localpref 100
                          AS path: ?, validation-state: unverified
                        >  to 192.168.10.10 via ge-0/0/0.11
    
    vrfx-a.inet.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
    + = Active Route, - = Last Active, * = Both
    
    172.30.30.0/24     *[EVPN/170] 00:09:56
                        >  to 192.168.10.10 via ge-0/0/0.11
    172.30.100.1/32    *[EVPN/170] 00:09:56
                        >  to 192.168.10.10 via ge-0/0/0.11
    172.30.100.2/32    *[Direct/0] 00:14:31
                        >  via lo0.100
    
    vrfx-b.inet.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
    + = Active Route, - = Last Active, * = Both
    
    192.168.2.0/24     *[EVPN/170] 00:08:41
                        >  to 192.168.10.10 via ge-0/0/0.11
    192.168.100.1/32   *[EVPN/170] 00:08:41
                        >  to 192.168.10.10 via ge-0/0/0.11
    192.168.100.2/32   *[Direct/0] 00:14:31
                        >  via lo0.101
    
    .....
    bgp.evpn.0: 6 destinations, 6 routes (6 active, 0 holddown, 0 hidden)
    + = Active Route, - = Last Active, * = Both
    
    5:65000:95001::0::172.30.30.0::24/248               
                       *[BGP/170] 00:33:41, MED 0, localpref 100, from 10.11.10.1
                          AS path: ?, validation-state: unverified
                        >  to 192.168.10.10 via ge-0/0/0.11
    5:65000:95001::0::172.30.100.1::32/248               
                       *[BGP/170] 00:33:41, MED 0, localpref 100, from 10.11.10.1
                          AS path: ?, validation-state: unverified
                        >  to 192.168.10.10 via ge-0/0/0.11
    5:65000:95001::0::172.30.100.2::32/248               
                       *[EVPN/170] 00:03:41
                           Indirect
    5:65000:95002::0::192.168.2.0::24/248               
                       *[BGP/170] 00:32:26, MED 0, localpref 100, from 10.11.10.1
                          AS path: ?, validation-state: unverified
                        >  to 192.168.10.10 via ge-0/0/0.11
    5:65000:95002::0::192.168.100.1::32/248               
                       *[BGP/170] 00:32:26, MED 0, localpref 100, from 10.11.10.1
                          AS path: ?, validation-state: unverified
                        >  to 192.168.10.10 via ge-0/0/0.11
    5:65000:95002::0::192.168.100.2::32/248               
                       *[EVPN/170] 00:03:07
                           Indirect
    
    vrfx-a.evpn.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
    + = Active Route, - = Last Active, * = Both
    
    5:65000:95001::0::172.30.30.0::24/248               
                       *[BGP/170] 00:00:05, MED 0, localpref 100, from 10.11.10.1
                          AS path: ?, validation-state: unverified
                        >  to 192.168.10.10 via ge-0/0/0.11
    5:65000:95001::0::172.30.100.1::32/248               
                       *[BGP/170] 00:00:05, MED 0, localpref 100, from 10.11.10.1
                          AS path: ?, validation-state: unverified
                        >  to 192.168.10.10 via ge-0/0/0.11
    5:65000:95001::0::172.30.100.2::32/248               
                       *[EVPN/170] 00:03:41
                           Indirect
    
    vrfx-b.evpn.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
    + = Active Route, - = Last Active, * = Both
    
    5:65000:95002::0::192.168.2.0::24/248               
                       *[BGP/170] 00:00:05, MED 0, localpref 100, from 10.11.10.1
                          AS path: ?, validation-state: unverified
                        >  to 192.168.10.10 via ge-0/0/0.11
    5:65000:95002::0::192.168.100.1::32/248               
                       *[BGP/170] 00:00:05, MED 0, localpref 100, from 10.11.10.1
                          AS path: ?, validation-state: unverified
                        >  to 192.168.10.10 via ge-0/0/0.11
    5:65000:95002::0::192.168.100.2::32/248               
                       *[EVPN/170] 00:03:07
                           Indirect

    Everything looks ok in the routing table – as you can see only in the inet.0 table, it is worth turning on the appropriate filtering of broadcast prefixes in BGP for the loopback from the T0 side, but this is cosmetic.

  3. favorite ping test between loopback interfaces
    1. VRF A
      root> ping 192.168.100.1 routing-instance vrfx-b    
      PING 192.168.100.1 (192.168.100.1): 56 data bytes
      64 bytes from 192.168.100.1: icmp_seq=0 ttl=64 time=1.266 ms
      64 bytes from 192.168.100.1: icmp_seq=1 ttl=64 time=1.429 ms
      64 bytes from 192.168.100.1: icmp_seq=2 ttl=64 time=1.347 ms
      64 bytes from 192.168.100.1: icmp_seq=3 ttl=64 time=1.386 ms
      64 bytes from 192.168.100.1: icmp_seq=4 ttl=64 time=1.698 ms
      64 bytes from 192.168.100.1: icmp_seq=5 ttl=64 time=1.530 ms
      64 bytes from 192.168.100.1: icmp_seq=6 ttl=64 time=1.450 ms
      ^C
      --- 192.168.100.1 ping statistics ---
      7 packets transmitted, 7 packets received, 0% packet loss
      round-trip min/avg/max/stddev = 1.266/1.444/1.698/0.129 ms
    2. VRF B
      root> ping 172.30.100.1 routing-instance vrfx-a  
      PING 172.30.100.1 (172.30.100.1): 56 data bytes
      64 bytes from 172.30.100.1: icmp_seq=0 ttl=64 time=1.325 ms
      64 bytes from 172.30.100.1: icmp_seq=1 ttl=64 time=1.249 ms
      64 bytes from 172.30.100.1: icmp_seq=2 ttl=64 time=1.390 ms
      ^C
      --- 172.30.100.1 ping statistics ---
      3 packets transmitted, 3 packets received, 0% packet loss
      round-trip min/avg/max/stddev = 1.249/1.321/1.390/0.058 ms

NSX-T side verification

On the NSX side, we check from the CLI. We log in from the EDGE VM on which T0 is running.

  1. We check which id is running SR for T0
    dc1edge03> get logical-router
    Logical Router
    UUID                                   VRF    LR-ID  Name                              Type                        Ports  
    736a80e3-23f6-5a2d-81d6-bbefb2786666   0      0                                        TUNNEL                      4      
    f9414cef-e3e3-405a-a7f8-55f349e08f22   1      1026   SR-VRF-VRF-A                      VRF_SERVICE_ROUTER_TIER0    6      
    11ab6c6f-222b-434f-b25b-aad51dadf2ed   2      1027   SR-VRF-VRF-B                      VRF_SERVICE_ROUTER_TIER0    6      
    987450ee-042c-41aa-9d0d-d4fcabd44b36   3      25     SR-R-T1-VRF-A                     SERVICE_ROUTER_TIER1        5      
    06d10402-2066-4e4f-8243-6679d1bcb6b3   4      14     SR-R-T0-MPBGP                     SERVICE_ROUTER_TIER0        7      
    f2ca25ae-49a3-4b12-a91e-c5c9908bb6c9   6      22     DR-VRF-VRF-A                      VRF_DISTRIBUTED_ROUTER_TIER0 4      
    c43a9c6d-b207-4022-b43c-e52728eb1ab5   7      24     DR-R-T1-VRF-A                     DISTRIBUTED_ROUTER_TIER1    4      
    304d5182-a40c-486a-a1e3-6d35275951fd   8      27     SR-R-T1-VRF-B                     SERVICE_ROUTER_TIER1        5      
    758b42b3-8a4c-452f-91ad-6baf97188879   9      23     DR-VRF-VRF-B                      VRF_DISTRIBUTED_ROUTER_TIER0 4      
    6204d89c-ec0e-40a9-9824-33adbbb41bff   10     26     DR-R-T1-VRF-B                     DISTRIBUTED_ROUTER_TIER1    4
  2. go to VRF 4 (SR T0) and check BGP status
    dc1edge03(tier0_sr)> get bgp neighbor summary 
    BFD States: NC - Not configured, AC - Activating,DC - Disconnected
                AD - Admin down, DW - Down, IN - Init,UP - Up
    BGP summary information for VRF default for address-family: ipv4Unicast
    Router ID: 192.168.10.10  Local AS: 65000
    
    Neighbor                            AS          State Up/DownTime  BFD InMsgs  OutMsgs InPfx  OutPfx
    
    10.11.10.2                          65000       Estab 00:20:34     NC  9887    8712    0      0     
    192.168.10.1                        65000       Estab 00:20:34     NC  9869    8910    1      0     
    
    BFD States: NC - Not configured, AC - Activating,DC - Disconnected
                AD - Admin down, DW - Down, IN - Init,UP - Up
    BGP summary information for VRF default for address-family: l2VpnEvpn
    Router ID: 192.168.10.10  Local AS: 65000
    
    Neighbor                            AS          State Up/DownTime  BFD InMsgs  OutMsgs InPfx  OutPfx
    
    10.11.10.2                          65000       Estab 00:20:34     NC  9887    8712    0      0     
    192.168.10.1                        65000       Estab 00:20:34     NC  9869    8910    0      0
  3. check status of BGP EVPN – get bgp evpn
    dc1edge03(tier0_sr)> get bgp evpn 
    BGP table version is 3, local router ID is 192.168.10.10
    Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
    Origin codes: i - IGP, e - EGP, ? - incomplete
    EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
    EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
    EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
    EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]
    
       Network          Next Hop            Metric LocPrf Weight Path
    Route Distinguisher: 65000:95001
    *> [5]:[0]:[24]:[172.30.30.0]
                        10.11.10.1               0         32768 ?
    *> [5]:[0]:[32]:[172.30.100.1]
                        10.11.10.1               0         32768 ?
    *>i[5]:[0]:[32]:[172.30.100.2]
                        10.11.10.2                    100      0 i
    Route Distinguisher: 65000:95002
    *> [5]:[0]:[24]:[192.168.2.0]
                        10.11.10.1               0         32768 ?
    *> [5]:[0]:[32]:[192.168.100.1]
                        10.11.10.1               0         32768 ?
    *>i[5]:[0]:[32]:[192.168.100.2]
                        10.11.10.2                    100      0 i
    
    Displayed 6 prefixes (6 paths)

    Some other helpful commands to verify from the T0 router (SR)

    Displays all available VRFs with assigned VNI get bgp evpn vni 
    Displays the BGP routing table by RD get bgp evpn overlay
    Shows configured VRFs that are active in EVPN get evpn vni

Posts created 126

Begin typing your search term above and press enter to search. Press ESC to cancel.

Back To Top