NSX for vSphere: Controller “Connections” and “VTEPs”

In many NSX training classes I heard the same question come up: “what do the Connections and VTEPs numbers represent in show control-cluster logical-switches vni XXX command output?”

nsx-controller # show control-cluster logical-switches vni 5001
 VNI      Controller      BUM-Replication ARP-Proxy Connections VTEPs
 5001     192.168.110.202 Enabled         Enabled   6           4

As training lab exercises build on as well as time passes, the numbers above also keep on changing. What is going on there?

Let’s shed some light on that part of NSX-v.

You can see from the command syntax above that it was ran on a Controller node; in this case the one that currently has master role for the VNI 5001. The output indicates that there are 6 connections to the Controller, and there are 4 VTEPs active.

The following diagram shows the environment this output was taken from (click to pop up in full size):

Sample setup

VTEPs

We know that ESXi host joins a VNI in two cases:

  1. When a VM running on that host connects to VNI’s dvPg and its vNIC transitions into “Link Up” state; and
  2. When DLR kernel module on that host needs to route traffic to a VM on that VNI that’s running on a different host.

From the output above we see there are 4 VTEPs that have joined the VNI 5001. Which ones are they?

nsx-controller # show control-cluster logical-switches vtep-table 5001
 VNI      IP              Segment         MAC               Connection-ID
 5001     192.168.250.53  192.168.250.0   00:50:56:67:d9:91 1845
 5001     192.168.250.52  192.168.250.0   00:50:56:64:f4:25 1843
 5001     192.168.250.51  192.168.250.0   00:50:56:66:e2:ef 7
 5001     192.168.150.51  192.168.150.0   00:50:56:60:bc:e9 3

Looking at our diagram, we can account for three VTEPs of the ESXi hosts with VMs web1 (192.168.250.51), web2 (192.168.250.53), and LB (192.168.150.51). The last remaining VTEP (192.168.250.52) is of the host esxcomp–02a, which has no VM on VNI 5001.

It, however, has VM app1, which provides application services to web frontend running on web1/web2. DLR on that host routes traffic from app1 back to web1/web2, and thus needs to reach them on VNI 5001, making the host join that VNI.

Connections

VTEPs out of the way, let’s have a look at Connections. According to the first command output, our Controller has 6 Connections. We can see which ones they are by using the following command:

nsx-controller # show control-cluster logical-switches connection-table 5001
 Host-IP         Port  ID
 192.168.110.52  15426 3
 192.168.210.51  48515 7
 192.168.210.56  48445 1845
 192.168.110.51  19833 1844
 192.168.210.52  17073 1843
 192.168.210.57  51467 1842

These Host-IP addresses are not VTEPs; rather they are of vmk0 interfaces, because connections between ESXi hosts and Controllers are performed via management network. Port here is the ephemeral TCP port allocated by ESXi host’s IP stack when the host was establishing connection with the Controller. Looking for Controller network connections on one of the hosts allows us to match that port number:

~ # esxcli network ip connection list | grep 1234
tcp 0 0 192.168.210.51:45340 192.168.110.203:1234 ESTABLISHED 5014686 newreno netcpa-worker
tcp 0 0 192.168.210.51:28298 192.168.110.201:1234 ESTABLISHED 5014686 newreno netcpa-worker
tcp 0 0 192.168.210.51:48515 192.168.110.202:1234 ESTABLISHED 9184583 newreno netcpa-worker

We can see port number 48515 matching with the this host’s vmk0 IP address of 192.168.210.51 from the connection-table command above.

Looks like all 6 hosts in our environment have connections open to the VNI 5001’s master controller. The question remains, however – why?

The answer to that lies with DLR. Remember the reason #2 for a host to join the VNI? DLR needs to be able to start routing traffic at a drop of a hat, so each host where DLR is configured with a VXLAN LIF needs to be ready for that. This readiness includes the following things:

  1. Knowing the Master Controller for that VNI;
  2. Having an open connection to that Master Controller;
  3. Having a local copy of that VNI’s VTEP table (to save time requesting it from the Controller)

Based on the commands above, we know that the host esx–01a has no VMs on VNI 5001 (and isn’t a VTEP for it), but has an active connection to the Controller. We can see that it also has a list of VTEPs for that VNI:

~ # esxcli network vswitch dvs vmware vxlan network vtep list --vds-name Mgmt_Edge_VDS --vxlan-id=5001
 IP              Segment ID     Is MTEP
 --------------  -------------  -------
 192.168.150.51  192.168.150.0    false
 192.168.250.53  192.168.250.0     true
 192.168.250.51  192.168.250.0    false
 192.168.250.52  192.168.250.0    false

Confirmation

To prove our understanding, we can drop the DLR (there’s only one in our playpen environment), and see what happens to the Connections and VTEPs:

nsx-controller # show control-cluster logical-switches vni 5001
 VNI      Controller      BUM-Replication ARP-Proxy Connections VTEPs
 5001     192.168.110.202 Enabled         Enabled   3           3

As expected, now we can only see three, for the ESXi hosts that have active VMs on our VNI. The host esxcomp–02a is no longer there since there’s no DLR and no ability to route from app1 back to web1/web2. We will also find that other hosts will have dropped connections to the Controller, and no longer have a VTEP table for that VNI:

~ # esxcli network ip connection list | grep 1234
tcp 0 0 192.168.210.51:45340 192.168.110.203:1234 ESTABLISHED 5014686 newreno netcpa-worker
tcp 0 0 192.168.210.51:28298 192.168.110.201:1234 ESTABLISHED 5014686 newreno netcpa-worker
~ # esxcli network vswitch dvs vmware vxlan network vtep list –vds-name Mgmt_Edge_VDS –vxlan-id=5001
Unable to find VTEP with specified parameter

Odds and ends

ESXi Host’s VNI membership caused by DLR kernel module sending traffic on that VNI is “on demand”, and expires shortly (slightly over 10 minutes) after when there’s no more traffic. This process may cause the number of VTEPs seen by the Controller to go up and down with time. On the other hand, VNI membership due to running / connected VM does not expire until the VM brings down vNIC Link, vNIC is disconnected from the VXLAN dvPg, or VM is shut down or moved to another host.

When NSX is deployed with a Controller Cluster, each ESXi host prepared for NSX will establish and keep a permanent connection to at least one of the Controllers. This Controller is selected by the netcpa userworld module on the host at random from the list of the Controllers received from NSX Manager via the message bus connection. If the selected Controller goes down, netcpa will cycle through the list until it manages to connect.

Hopefully, the above answers that particular question for good. 🙂

Happy festive season! 🙂

Advertisements

About Dmitri Kalintsev

Some dude with a blog and opinions ;) View all posts by Dmitri Kalintsev

6 responses to “NSX for vSphere: Controller “Connections” and “VTEPs”

  • Roie Ben Haim

    Very useful and informative, love reading your blog

  • NSX-v Troubleshooting L2 Connectivity | VMware Professional Services

    […] His name is Dmitri Kalintsev. link to his blog: NSX for vSphere: Controller “Connections” and “VTEPs” […]

  • Michael

    Do the host-local VTEP, ARP and MAC tables received from controller have expiration time? in case management cluster gets disconnected from compute will compute hosts lose connectivity with each other after some time?

    • Dmitri Kalintsev

      Michael,

      As far as I know, VTEP table entries don’t have expiry timers; they are only updated by datapath learning (host’s VXLAN kernel module, vdl2) and Controller messages. VTEP table *may* get flushed if host’s control plane agent (netcpa) gets restarted – I haven’t checked.

      ARP suppression entries expire in 180 seconds. DLR ARP enties expire in 600 seconds.

      MAC entries expire at around 200 seconds (see my post on VXLAN control plane where I covered it).

      When you’re talking about management cluster, do you mean Controller Cluster? If yes, then hosts that lost Controller connection will keep on using VTEP entries they already have (which may get out of date if other hosts join/leave VNIs).

      ARP and MAC tables are less important – for ARP it simply means that VMs ARP queries will get flooded, and for MACs VXLAN will revert to flooding to all VTEPs on given VNI, and datapath learning (again, please refer to the VXLAN control plane post).

      HTH.

  • NSX-v: Follow the IP/MAC address | SneakU

    […] to Dmitri Kalintsev and this post, we know that we can use the following command to display the connection-table, which will show the […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: