In many NSX training classes I heard the same question come up: “what do the Connections and VTEPs numbers represent in
show control-cluster logical-switches vni XXX command output?”
nsx-controller # show control-cluster logical-switches vni 5001 VNI Controller BUM-Replication ARP-Proxy Connections VTEPs 5001 192.168.110.202 Enabled Enabled 6 4
As training lab exercises build on as well as time passes, the numbers above also keep on changing. What is going on there?
Let’s shed some light on that part of NSX-v.
You can see from the command syntax above that it was ran on a Controller node; in this case the one that currently has master role for the VNI 5001. The output indicates that there are 6 connections to the Controller, and there are 4 VTEPs active.
The following diagram shows the environment this output was taken from (click to pop up in full size):
We know that ESXi host joins a VNI in two cases:
- When a VM running on that host connects to VNI’s dvPg and its vNIC transitions into “Link Up” state; and
- When DLR kernel module on that host needs to route traffic to a VM on that VNI that’s running on a different host.
From the output above we see there are 4 VTEPs that have joined the VNI 5001. Which ones are they?
nsx-controller # show control-cluster logical-switches vtep-table 5001 VNI IP Segment MAC Connection-ID 5001 192.168.250.53 192.168.250.0 00:50:56:67:d9:91 1845 5001 192.168.250.52 192.168.250.0 00:50:56:64:f4:25 1843 5001 192.168.250.51 192.168.250.0 00:50:56:66:e2:ef 7 5001 192.168.150.51 192.168.150.0 00:50:56:60:bc:e9 3
Looking at our diagram, we can account for three VTEPs of the ESXi hosts with VMs web1 (192.168.250.51), web2 (192.168.250.53), and LB (192.168.150.51). The last remaining VTEP (192.168.250.52) is of the host esxcomp–02a, which has no VM on VNI 5001.
It, however, has VM app1, which provides application services to web frontend running on web1/web2. DLR on that host routes traffic from app1 back to web1/web2, and thus needs to reach them on VNI 5001, making the host join that VNI.
VTEPs out of the way, let’s have a look at Connections. According to the first command output, our Controller has 6 Connections. We can see which ones they are by using the following command:
nsx-controller # show control-cluster logical-switches connection-table 5001 Host-IP Port ID 192.168.110.52 15426 3 192.168.210.51 48515 7 192.168.210.56 48445 1845 192.168.110.51 19833 1844 192.168.210.52 17073 1843 192.168.210.57 51467 1842
These Host-IP addresses are not VTEPs; rather they are of vmk0 interfaces, because connections between ESXi hosts and Controllers are performed via management network. Port here is the ephemeral TCP port allocated by ESXi host’s IP stack when the host was establishing connection with the Controller. Looking for Controller network connections on one of the hosts allows us to match that port number:
~ # esxcli network ip connection list | grep 1234 tcp 0 0 192.168.210.51:45340 192.168.110.203:1234 ESTABLISHED 5014686 newreno netcpa-worker tcp 0 0 192.168.210.51:28298 192.168.110.201:1234 ESTABLISHED 5014686 newreno netcpa-worker tcp 0 0 192.168.210.51:48515 192.168.110.202:1234 ESTABLISHED 9184583 newreno netcpa-worker
We can see port number 48515 matching with the this host’s vmk0 IP address of 192.168.210.51 from the connection-table command above.
Looks like all 6 hosts in our environment have connections open to the VNI 5001’s master controller. The question remains, however – why?
The answer to that lies with DLR. Remember the reason #2 for a host to join the VNI? DLR needs to be able to start routing traffic at a drop of a hat, so each host where DLR is configured with a VXLAN LIF needs to be ready for that. This readiness includes the following things:
- Knowing the Master Controller for that VNI;
- Having an open connection to that Master Controller;
- Having a local copy of that VNI’s VTEP table (to save time requesting it from the Controller)
Based on the commands above, we know that the host esx–01a has no VMs on VNI 5001 (and isn’t a VTEP for it), but has an active connection to the Controller. We can see that it also has a list of VTEPs for that VNI:
~ # esxcli network vswitch dvs vmware vxlan network vtep list --vds-name Mgmt_Edge_VDS --vxlan-id=5001 IP Segment ID Is MTEP -------------- ------------- ------- 192.168.150.51 192.168.150.0 false 192.168.250.53 192.168.250.0 true 192.168.250.51 192.168.250.0 false 192.168.250.52 192.168.250.0 false
To prove our understanding, we can drop the DLR (there’s only one in our playpen environment), and see what happens to the Connections and VTEPs:
nsx-controller # show control-cluster logical-switches vni 5001 VNI Controller BUM-Replication ARP-Proxy Connections VTEPs 5001 192.168.110.202 Enabled Enabled 3 3
As expected, now we can only see three, for the ESXi hosts that have active VMs on our VNI. The host esxcomp–02a is no longer there since there’s no DLR and no ability to route from app1 back to web1/web2. We will also find that other hosts will have dropped connections to the Controller, and no longer have a VTEP table for that VNI:
~ # esxcli network ip connection list | grep 1234 tcp 0 0 192.168.210.51:45340 192.168.110.203:1234 ESTABLISHED 5014686 newreno netcpa-worker tcp 0 0 192.168.210.51:28298 192.168.110.201:1234 ESTABLISHED 5014686 newreno netcpa-worker
~ # esxcli network vswitch dvs vmware vxlan network vtep list –vds-name Mgmt_Edge_VDS –vxlan-id=5001 Unable to find VTEP with specified parameter
Odds and ends
ESXi Host’s VNI membership caused by DLR kernel module sending traffic on that VNI is “on demand”, and expires shortly (slightly over 10 minutes) after when there’s no more traffic. This process may cause the number of VTEPs seen by the Controller to go up and down with time. On the other hand, VNI membership due to running / connected VM does not expire until the VM brings down vNIC Link, vNIC is disconnected from the VXLAN dvPg, or VM is shut down or moved to another host.
When NSX is deployed with a Controller Cluster, each ESXi host prepared for NSX will establish and keep a permanent connection to at least one of the Controllers. This Controller is selected by the netcpa userworld module on the host at random from the list of the Controllers received from NSX Manager via the message bus connection. If the selected Controller goes down, netcpa will cycle through the list until it manages to connect.
Hopefully, the above answers that particular question for good. 🙂
Happy festive season! 🙂