In a previous post I showed how to configure EVPN in NSX-T 3.0 with Juniper vMX, where the MP-BGP session originated from an Edge node through the T0 Gateway and VRFs were terminated on T0 and connected to T1. Today I’ll cover the next step in the evolution of that feature in VCF 9.1 — Distributed Transit Gateway with EVPN/VXLAN, where Edge nodes disappear from the picture entirely and N/S traffic flows directly from each ESX host through a VXLAN tunnel to an external BGW.
Below is the full configuration from my test environment using VyOS as the BGW and four ESXi hosts in the homelab.int domain.
What is DTGW with EVPN/VXLAN
DTGW (Distributed Transit Gateway) is a new option for connecting VCF to the physical network, available since VCF 9.0. VCF 9.1 adds support for EVPN/VXLAN as the uplink type instead of a traditional VLAN — the same BGP EVPN Type-5 mechanism familiar from physical fabrics like Arista, Cisco Nexus, and Juniper.
In short, the differences compared to the old NSX-T 3.0 approach:
| NSX-T 3.0 / EVPN on T0 | VCF 9.1 / DTGW EVPN/VXLAN |
|---|---|
| BGP EVPN originates from Edge node | BGP EVPN originates from Route Controller (VNA) |
| Traffic: VM → Edge → BGW | Traffic: VM → Host TEP → BGW (no Edge) |
| VXLAN tunnels terminate on Edge | VXLAN tunnels terminate on each ESX host (TEP) |
| VRFs on T0 Gateway | VRFs managed by DTGW + Route Controller |
| Edge node lifecycle required | No Edge — no Edge lifecycle |
The BGP EVPN Type-5 mechanism, Route Distinguisher, and Route Target work identically to what I described in the previous NSX-T EVPN post — only the tunnel termination point and management model change.
Requirements and limitations
| Requirements | Limitations in VCF 9.1 |
|---|---|
| VCF 9.1 / NSX 9.1 | No N-S Services (SNAT, LB) — planned for a future VCF release |
| BGW supporting BGP EVPN Type-5 (MP-BGP) | L3 only (Route Type-5) — no L2 extension to the fabric |
| Underlay MTU min. 1600 (VXLAN overhead ~50 B) | Access Mode: Private requires manual NAT on the BGW side |
| DVS port group required for the RC BGP peering network | VyOS VRF table ID must be <1–200 if you plan to use NAT |
Test environment
| Element | Value | Note |
|---|---|---|
| Hosty ESXi | ESXi11 – ESXi14 |
domain homelab.int |
| VLAN TEP Host | 10.231.104.0/24 |
vmk Host TEP |
| VLAN BGP Peering | 10.231.106.0/24 |
eth2.506 on VyOS |
| Peer VyOS (BGW) | 10.231.106.1 |
BGP update-source |
| Peer RC (BGP VIP) | 10.231.106.10 |
Route Controller floating IP |
| ASN VyOS | 64515 |
BGP system-as |
| ASN Route Controller | 64520 |
remote-as in VyOS |
| L3VNI / VNI | 65000 |
tenant VRF identifier |
| VRF (VyOS) | test |
table 65000 |
| Subnet VM (External IP Block) | 10.231.200.0/24 |
VM addresses in Public mode |
| RD VyOS (global) | 64515:10 |
|
| RD VyOS (VRF) | 10.231.106.1:10 |
unique per-VRF |
| Export RT VyOS | 64515:10 |
|
| Import RT VyOS | 64515:10 + 64520:10 |
also imports RT from RC |
| Export RT RC | 64520:10 |
configured in NSX External Connectivity |
| Import RT RC | 64515:10 |
imports RT from VyOS |
Diagram
Connectivity check before configuration
Before deploying the RC it is worth verifying that the ESXi hosts can reach the address that will be used as the VXLAN source-address on the VyOS side. Ping from vmk0 and vmk1 to the BGW address:
[root@ESXi11:~] ping -I vmk10 10.231.106.1 -S vxlan
PING 10.231.106.1 (10.231.106.1): 56 data bytes
64 bytes from 10.231.106.1: icmp_seq=0 ttl=64 time=0.425 ms
64 bytes from 10.231.106.1: icmp_seq=1 ttl=64 time=0.424 ms
64 bytes from 10.231.106.1: icmp_seq=2 ttl=64 time=0.394 ms
3 packets transmitted, 3 packets received, 0% packet loss
[root@ESXi11:~] ping -I vmk11 10.231.106.1 -S vxlan
PING 10.231.106.1 (10.231.106.1): 56 data bytes
64 bytes from 10.231.106.1: icmp_seq=0 ttl=64 time=0.748 ms
64 bytes from 10.231.106.1: icmp_seq=1 ttl=64 time=0.820 ms
64 bytes from 10.231.106.1: icmp_seq=2 ttl=64 time=0.340 ms
3 packets transmitted, 3 packets received, 0% packet lossNo response at this stage means VXLAN tunnels will not work even if the BGP session is Established. We set the MTU to 9000; VXLAN adds ~50 B of overhead so there is plenty of headroom.
VyOS configuration (BGW)
Interfaces — VXLAN, bridge
# trunk VLAN on eth2, MTU 9000 (underlay)
set interfaces ethernet eth2 mtu '9000'
set interfaces ethernet eth2 vif 506 address '10.231.106.1/24'
set interfaces ethernet eth2 vif 504 address '10.231.104.1/24'
# VXLAN — VNI 65000 = tenant L3VNI
# source-address = BGP peering address
set interfaces vxlan vxlan65000 mtu '1500'
set interfaces vxlan vxlan65000 port '4789'
set interfaces vxlan vxlan65000 source-address '10.231.106.1'
set interfaces vxlan vxlan65000 vni '65000'
# bridge connects VXLAN to VRF "test"
set interfaces bridge br1 address '10.231.251.1/24'
set interfaces bridge br1 member interface vxlan65000
set interfaces bridge br1 vrf 'test'Global BGP and Route Controller session
set protocols bgp system-as '64515'
set protocols bgp parameters router-id '10.231.106.1'
set protocols bgp parameters bestpath as-path multipath-relax
set protocols bgp address-family ipv4-unicast redistribute connected
set protocols bgp address-family ipv4-unicast redistribute static
set protocols bgp address-family ipv4-unicast import vrf 'test'
set protocols bgp address-family l2vpn-evpn advertise ipv4 unicast
set protocols bgp address-family l2vpn-evpn advertise-all-vni
set protocols bgp address-family l2vpn-evpn advertise-default-gw
set protocols bgp address-family l2vpn-evpn advertise-svi-ip
set protocols bgp address-family l2vpn-evpn default-originate ipv4
# sesja BGP z Route Controllerem (BGP VIP)
set protocols bgp neighbor 10.231.106.10 remote-as '64520'
set protocols bgp neighbor 10.231.106.10 update-source '10.231.106.1'
set protocols bgp neighbor 10.231.106.10 address-family ipv4-unicast default-originate
set protocols bgp neighbor 10.231.106.10 address-family ipv4-vpn route-server-client
set protocols bgp neighbor 10.231.106.10 address-family l2vpn-evpn soft-reconfiguration inboundA few notes on the RC neighbor parameters:
default-originate— VyOS advertises the default route to the RC which then distributes it to ESX hostssoft-reconfiguration inbound— reset BGP policy without dropping the session, useful during testingroute-server-client— facilitates route propagation between BGP clients through the RC
VRF “test”
Note — VyOS requires the table ID to be in the range 1–200 if you plan to use connection-mark for NAT from within the VRF. In this example I use table '65000' which works fine for routing, but consider changing it to e.g. 100 if you plan to add NAT later.
set vrf name test table '65000'
set vrf name test vni '65000'
set vrf name test protocols bgp system-as '64515'
set vrf name test protocols bgp parameters router-id '10.231.106.1'
set vrf name test protocols bgp address-family ipv4-unicast import vrf 'default'
set vrf name test protocols bgp address-family ipv4-unicast redistribute connected
set vrf name test protocols bgp address-family ipv4-unicast redistribute static
# RD and RT per-VRF — different from global!
set vrf name test protocols bgp address-family l2vpn-evpn rd '10.231.106.1:10'
set vrf name test protocols bgp address-family l2vpn-evpn route-target export '64515:10'
set vrf name test protocols bgp address-family l2vpn-evpn route-target import '64520:10'
set vrf name test protocols bgp address-family l2vpn-evpn route-target import '64515:10'
set vrf name test protocols bgp address-family l2vpn-evpn vni 65000
set vrf name test protocols bgp address-family l2vpn-evpn advertise ipv4 unicast
set vrf name test protocols bgp address-family l2vpn-evpn advertise-default-gw
set vrf name test protocols bgp address-family l2vpn-evpn advertise-svi-ipRoute Target logic:
| Side | Export RT | Import RT | Effect |
|---|---|---|---|
| VyOS (BGW) | 64515:10 |
64515:10, 64520:10 |
VyOS imports routes from RC and its own |
| Route Controller (NSX) | 64520:10 |
64515:10 |
RC imports routes from VyOS (default route and external networks) |
VyOS configuration verification — VNI and bridge
After finishing the VyOS configuration, verify that the VNI is active and the bridge is properly set up:
vyos@vyos:~$ show bgp l2vpn evpn vni
Advertise Gateway Macip: Enabled
Advertise SVI Macip: Enabled
Advertise All VNI flag: Enabled
BUM flooding: Head-end replication
VXLAN flooding: Enabled
Number of L2 VNIs: 1
Number of L3 VNIs: 1
Flags: * - Kernel
VNI Type RD Import RT Export RT Tenant VRF
65000 L2 10.231.106.1:2 64515:10 64515:10 default
* 65000 L3 10.231.106.1:10 64515:10, ... 64515:10 test
# * = Kernel = active in kernel table (L3 VNI)
# L2 VNI without asterisk = bridge with no VPC subnet assigned (normal before adding a VM)vyos@vyos:~$ show evpn vni 65000
VNI: 65000
Type: L3
Tenant VRF: test
Vlan: 1
Bridge: br1
Local Vtep Ip: 10.231.106.1
Vxlan-Intf: vxlan65000
SVI-If: br1
State: Up
VNI Filter: none
System MAC: c2:8f:db:73:82:3c
Router MAC: c2:8f:db:73:82:3c
L2 VNIs:State: Up and Local Vtep Ip matching the source-address of the vxlan65000 interface — this is the address that will appear as the VTEP in BGP EVPN Type-5 advertisements to the RC.
vyos@vyos:~$ show bridge br1 detail
Interface br1 is up, line protocol is up
Link ups: 13 last: 2026/05/07 10:28:54.00
Link downs: 14 last: 2026/05/07 10:28:54.00
vrf: test
index 17 metric 0 mtu 1500 speed 0 txqlen 1000
flags: <UP,LOWER_UP,BROADCAST,RUNNING,MULTICAST>
Type: Ethernet
HWaddr: c2:8f:db:73:82:3c # this MAC == Router MAC in EVPN
inet 10.231.251.1/24
Interface Type Bridge
Interface Slave Type Vrf
Bridge VLAN-aware: no
protodown: offThe MAC address of bridge br1 (c2:8f:db:73:82:3c) is used as the Router MAC in RT-5 advertisements. This is why the RD on the VyOS side and on the RC side must be different — each side needs its own unique values.
NSX / VCF configuration
Step 1 — Deploy the Route Controller
Navigate to NSX Manager –> Network –> EVPN –> Route Controllers and click Add Route Controller.
Fill in the details:
- Route Controller Name: evpn-test
- Node Form Factor: small
Click Add and fill in the Node 1 details — Management Network information for the virtual appliance:
- Management IP, Gateway, DNS — values from your mgmt network
- Datastore, Cluster, Host — select from your vCenter
After filling in the node details click Next (I use a single node in the lab). On the next screen configure the BGP interface:
- Port Group — DVS port group for VLAN 506 (BGP peering), must exist before deployment
- Node 1 IP / Node 2 IP — node addresses on the BGP network
- BGP Floating IP — 10.231.106.10 — active RC VIP, this address goes into VyOS as the
neighbor - Prefix length —
/24
After clicking Finish the RC is deployed automatically. Wait 5–10 minutes, then verify the status:
Click on BGP Neighbors:
An Idle state on one node is normal — the active RC shows Established, the standby shows Idle. If the active node also shows Idle or Active, do Disable –> Enable on the BGP Neighbor in the NSX UI and wait a moment.
Verify on the VyOS side — State/PfxRcd shown as a number (not “Active” / “Connect”) = session established:
vyos@vyos:~$ show bgp l2vpn evpn summary
BGP router identifier 10.231.106.1, local AS number 64515 VRF default vrf-id 0
Neighbor V AS MsgRcvd MsgSent TblVer Up/Down State/PfxRcd PfxSnt
10.231.106.10 4 64520 2991 3024 15 22:20:56 2 16 N/A
# State/PfxRcd as a number — session Established, RC receives 2 prefixes from VyOS
# other neighbors (10.231.107.11, .12 etc.) — Active = other segments, not this EVPN/VXLAN setupStep 2 — External Connectivity (Distributed VXLAN Connection)
Navigate to Network –> VPC Connectivity –> External Connectivity and click Add External Connectivity.
Fill in the form:
- Type —
Distributed VXLAN Connection - Route Controller — evpn-test (deployed in Step 1)
- EVPN L3 VNI — 65000 — must match
vni 65000in VyOS - Route Distinguisher — 10.231.106.10:65000 — BGP VIP : VNI; must differ from the VyOS RD (
10.231.106.1:10) - Import Route Target — 64515:10 — import routes exported by VyOS
- Export Route Target — 64520:10 — VyOS must import this
The creation task completes within a few seconds.
Step 3 — Configure DTGW
Navigate to Network –> VPC Connectivity –> Transit Gateway. You can edit the default TGW or add a new one.
In the TGW edit form:
- Connectivity (Uplink Type) — change to
Distributed VXLAN - External Connectivity — select the record matching our RC
After saving, open the VPC Connectivity Profile associated with the TGW and edit it. Define the address pools:
- External IP Address Block — 10.231.200.0/24 — pool routable from the BGW, used to assign IPs to VMs in Public mode
- Private TGW IP Address Block — 172.26.0.0/16 — private pool within the TGW
Creating a VPC and Subnet
VPC
In vSphere Client navigate to Virtual Private Clouds –> New Virtual Private Cloud.
- Name — test-vpc
- Private VPC IP CIDR — 192.168.0.0/16
- Advanced Settings –> Connectivity Profile — verify that the profile linked to the DTGW is selected
Optional — Default VPC Service Profile
Before creating a subnet you can edit the Default VPC Service Profile to have DHCP advertise DNS and NTP. Navigate to VPC –> Profiles –> VPC Service Profiles –> edit Default VPC Service Profile.
- DNS Server — 10.231.106.1 (VyOS)
- NTP Server — 10.231.106.1 (VyOS)
Subnet (Public mode)
In the VPC view click New Subnet…
- Subnet Name — public-sub-01
- VLAN Extension —
No(L2 extension feature, not applicable for EVPN/VXLAN) - Access Mode — Public
- IP Block — select from External IP Address Block 10.231.200.8/29
In Advanced Settings:
- VPC Gateway Connectivity —
Yes(default, leave as is) - DHCP — DHCP Server
Connecting a VM and verification
Connecting the vNIC to the VPC subnet
In the VM settings under the VPC Subnets tab select test and save.
Verification from the Guest OS
After powering on the VM verify the DHCP address and default gateway:
VyOS verification — BGP EVPN routes
After powering on the VM, /32 routes to each VM should appear in the VyOS VRF test routing table with next-hop = TEP of the ESX host running the VM:
vyos@vyos:~$ show bgp l2vpn evpn summary
BGP router identifier 10.231.106.1, local AS number 64515 VRF default vrf-id 0
RIB entries 9, using 1296 bytes of memory
Peers 1, using 29 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer Up/Down State/PfxRcd PfxSnt
10.231.106.10 4 64520 2991 3024 15 22:20:56 2 16 N/A
# State/PfxRcd = 2 — RC is advertising 2 prefixes (/32 for 2 VMs)
# PfxSnt = 16 — VyOS is advertising 16 prefixes (connected + default route)vyos@vyos:~$ show ip route vrf test
B>* 10.231.200.3/32 [20/0] via 10.231.104.3, br1 onlink # VM1 — TEP 10.231.104.3
B>* 10.231.200.11/32 [20/0] via 10.231.104.7, br1 onlink # VM2 — TEP 10.231.104.7
# After vMotion VM1 to another host:
# B>* 10.231.200.3/32 [20/0] via 10.231.104.X, br1 onlink — next-hop changes automaticallyA /32 entry with next-hop = ESXi TEP address confirms that the Route Controller is correctly reporting VM location. After a vMotion the next-hop changes automatically — the same GARP-to-RC mechanism as in NSX-T 3.0, just without the Edge node.
RT-5 details in BGP EVPN — mapping VMs to TEPs
Full BGP EVPN table on the VyOS side — two Route Distinguishers are visible: local VyOS and the RC:
vyos@vyos:~$ show bgp l2vpn evpn
BGP table version is 2, local router ID is 10.231.106.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 64520:10 # routes from RC (VMs in VPC)
*> [5]:[0]:[32]:[10.231.200.3] # /32 VM1
10.231.104.3 0 64520 i
RT:64520:10 ET:8 Rmac:00:50:56:6a:68:38
*> [5]:[0]:[32]:[10.231.200.11] # /32 VM2
10.231.104.7 0 64520 i
RT:64520:10 ET:8 Rmac:00:50:56:65:18:e5
Route Distinguisher: 10.231.106.1:10 # local VyOS routes
*> [5]:[0]:[0]:[0.0.0.0] # default route — advertised to RC
10.231.106.1 0 32768 ?
ET:8 RT:64515:10 Rmac:c2:8f:db:73:82:3c
*> [5]:[0]:[24]:[10.231.104.0] # TEP VLAN
10.231.106.1 0 32768 ?
ET:8 RT:64515:10 Rmac:c2:8f:db:73:82:3c
*> [5]:[0]:[24]:[10.231.106.0] # BGP peering VLAN
10.231.106.1 0 32768 ?
ET:8 RT:64515:10 Rmac:c2:8f:db:73:82:3c
# ... remaining connected networks
Displayed 16 out of 16 total prefixesEach /32 under RD 64520:10 is one VM in the NSX VPC. Next-hop (10.231.104.3, 10.231.104.7) = ESX host TEP. Router MAC (Rmac) = TEP MAC address. After a vMotion the RC updates the advertisement via GARP — the next-hop change is visible within a few seconds.
Note — the output shows RD 64520:10 (from the RC), not 10.231.106.10:65000. The Route Distinguisher on the RC side is set as ASN_RC:VNI — the value comes from the External Connectivity configuration in NSX.
ESX verification — nsxcli
[root@ESXi11:~] nsxcli
esxi11.homelab.int> get gateway
87b192f1-a8c4-40f7-b9e6-1e905aa259c8 Gateway (Logical Router) UUID
bdc694b7-d236-47c3-93b0-8d2ba4f04608 Gateway (Logical Router) UUID
esxi11.homelab.int> get gateway 87b192f1-a8c4-40f7-b9e6-1e905aa259c8 forwarding
Mon May 11 2026 UTC 12:04:05.454
Flags: [U: Up], [G: Gateway], [S1|S2|S3: TGW Private|Public|External Route Scope]
Network Gateway Type Interface UUID
===================================================================================
0.0.0.0/0 10.231.106.1 UGS3 2d08b262-...
10.231.200.8/29 100.64.0.1 UGS2 a56ae99c-...
100.64.0.0/31 0.0.0.0 UCI a56ae99c-...
# S3 = External Route Scope (from RC/gateway)
# S2 = Public Route Scope (VM subnet)Verification via net-vdr (ESX)
net-vdr is a low-level ESX command that shows the internal routing tables of the Distributed Router:
[root@esxi11:~] net-vdr -Il --brief
DR Id #Lifs #Routesv4 State DR UUID
------- ----- --------- ----- ---------
0x1 2 3 A 87b192f1-a8c4-40f7-b9e6-1e905aa259c8 (/transit-gateways/default)
0xa 3 4 A bdc694b7-d236-47c3-93b0-8d2ba4f04608 (/vpcs/test-vpc,default)
[root@esxi11:~] net-vdr -Rl 87b192f1-a8c4-40f7-b9e6-1e905aa259c8
DR 87b192f1-... Route Table
Legend: [S1|S2|S3: TGW Private|Public|External Route Scope]
Destination GenMask Gateway Flags Ref UpTime HitCount
0.0.0.0 0.0.0.0 10.231.106.1 UGS3 1 314380 28
10.231.200.8 255.255.255.248 100.64.0.1 UGS2 1 314380 39
100.64.0.0 255.255.255.254 0.0.0.0 UCI 1 314380 5
# S3 = default route from RC/BGW (10.231.106.1 = VyOS)
# S2 = Public subnet 10.231.200.8/29Ping test from ESX host to VM:
[root@ESXi11:~] ping -I vmk0 10.231.200.3
64 bytes from 10.231.200.3: icmp_seq=0 ttl=61 time=1.8 ms
# ttl=61 = 3 hops (VM -> ESX TEP -> VyOS -> ping source)Common issues
| Problem | Likely cause | Where to look |
|---|---|---|
| BGP Neighbor in Idle state | Wrong remote-as, no route to BGP VIP, MTU issue | show bgp neighbor 10.231.106.10, ping from update-source |
| No /32 in VRF after VM power-on | RT mismatch, wrong L3VNI, RC not receiving notifications | show bgp l2vpn evpn, logi NSX Manager |
| VM cannot ping outside | Default route not propagated to ESX host, missing NAT | nsxcli: get gateway forwarding, show ip route vrf test |
| RT-5 routes visible but traffic not flowing | Underlay MTU issue, VTEP unreachable, encapsulation problem | ping from ESXi to GW-VTEP, tcpdump -i vxlan65000 |
| BGP Active on VyOS side | RC still starting up, wrong BGP VIP | Disable –> Enable BGP Neighbor in NSX UI, wait 2–3 min |
Summary
The VCF side of the configuration is straightforward — Route Controller, External Connectivity, and DTGW are just a few screens in NSX Manager. The real challenge is the BGW side. VyOS works very well for testing thanks to its debugging capabilities through vtysh and FRR.
A few things worth keeping in mind:
- Route Target must be configured symmetrically — what the RC exports (64520:10), VyOS must import, and vice versa
- L3VNI must be identical on both sides
- Underlay MTU must account for the ~50 B VXLAN overhead
- The VXLAN interface source-address must be reachable from the ESX host TEPs
- Keep the VyOS VRF table ID in the range 1–200 if you plan to use NAT with connection-mark
