Today we’ll look at a couple of choices you’ll need to make when deploying Brocade’s HW VTEP, and then check if our configuration is correct before linking it with NSX-v next time.
In the previous post we’ve touched on where a HW VTEP can sit in your network topology-wise, as well as what components need to communicate for the overall solution to work.
The Connectivity Needs diagram in that post showed that we have two distinct IP routing domains: one supporting Management and Control plane, while another supporting Data Plane and, to an extent, bits of control plane related to HW VTEP – namely BFD sessions to Replication Nodes.
Management / Control Plane connectivity
As mentioned before, Brocade HW VTEP solution is built around VCS fabric. To support management access, nodes (rbridges) of the fabric maintain a VCS Virtual IP. At any given time it is owned by one of the nodes.
It is typical for this IP to belong to the Management VRF, and be on the same subnet as the Management interfaces of your fabric nodes.
Hardware Switch Controller (HSC) component of overlay-gateway subsystem on VCS uses Virtual IP for Management and Control plane communications, namely for talking to NSX-v Controllers. Therefore, you will need to make sure you have a route in your VCS’s Management VRF toward your NSX infrastructure Management network, in particular covering the IP addresses of your Controller Cluster.
After you’ve deployed and configured your VCS fabric, the first test to conduct is to check if your VCS Virtual IP can reach your Controllers.
Since we’re starting to run some commands now, it’s time to introduce our lab environment:
On it, we have:
- Our standard VC/NSX components – vCenter Server, NSXM, and 3 x Controllers in the top right corner;
- 4 x ESXi hosts, two of which will be configured as Replication Nodes, in the top left corner; and
- A VCS fabric with two rbridges (physical nodes) at the bottom.
Things are also connected to two IP networks: Management (dark gray) and VTEP (black); both with Router elements in them. Please refer to earlier posts on the meaning of this, especially the Router (Data) one.
With the above in mind, let’s check if our VCS fabric’s Virtual IP (172.24.86.100) can reach the Controllers (172.24.87.150–152). SSH into the VCS Virtual IP (which will connect us to the rbridge that is “Associated” with it), and run the following ping command:
VDX6740-101# ping 172.24.87.150 vrf mgmt-vrf src-addr 172.24.86.100 Type Control-c to abort PING 172.24.87.150 (172.24.87.150) from 172.24.86.100: 56 data bytes 64 bytes from 172.24.87.150: icmp_seq=0 ttl=63 time=0.946 ms 64 bytes from 172.24.87.150: icmp_seq=1 ttl=63 time=0.523 ms 64 bytes from 172.24.87.150: icmp_seq=2 ttl=63 time=0.457 ms 64 bytes from 172.24.87.150: icmp_seq=3 ttl=63 time=0.564 ms 64 bytes from 172.24.87.150: icmp_seq=4 ttl=63 time=2.745 ms --- 172.24.87.150 ping statistics --- 5 packets transmitted, 5 packets received, 0% packet loss round-trip min/avg/max/stddev = 0.457/1.047/2.745/0.866 ms
Ok, we can reach one of the Controllers, which means that our IP connectivity is working.
For extra credit, let’s see what happens if we try to do the same from an rbridge other than the “associated” one:
VDX6740-102# sh vcs virtual-ip Virtual IP : 172.24.86.100/24 Associated rbridge-id : 101 Oper-Status : up VDX6740-102# ping 172.24.87.150 vrf mgmt-vrf src-addr 172.24.86.100 Type Control-c to abort bind source address:: Cannot assign requested address
Yip, no worky (we tried from 102, while the Associated one is 101). This titbit will become relevant when figuring out why your Management/Control plane may not be working at a later stage, since VCS talks to NSX Controllers from this IP, and thus things like OVSDB would also be handled on the Associated rbridge.
Next up, we need to deal with the VTEP on our VCS, which is where it gets interesting.
Brocade HW VTEP implementation supports two configurations for interfaces backing the VTEP:
- A Loopback interface, one for each HW VTEP rbridge, using the same /32 IP address on all rbridges; or
- A Virtual Ethernet (Ve) interface on each rbridge, configured as a member of a vrrp-extended-group, with a virtual-ip and a virtual-mac.
This brings us to a choice we need to make: which way to go?
Both ways, according to my contacts in the know, deliver the same functionality; so there are no benefits / drawbacks to help us decide. Let’s turn to how things work then, and figure this one out.
Loopbacks are simpler from the configuration standpoint, and do not require a dynamic protocol, such as VRRP-E, to function. They are also simpler to script configuration for, since we’re using the same IP address on all rbridges.
On other hand, we still need to configure IP routing within our VCS fabric. While L2 space is “automagically flat” fabric-wide, from an L3 standpoint each rbridge is an independent element requiring appropriate L3 configuration in the form of VRFs, interfaces, and routes. A typical solution for L3 switches is to create and assign IP addresses to Virtual Ethernet interfaces (think an “SVI” in C-Vendor’s terms) associated with L2 domains (VLANs).
Before choosing where to go, it’s often useful to consider where we are starting from.
If we already have a functioning VCS fabric and just want to add the overlay-gateway function to it, we may already have needed IP routing set up and in place. In this case, configuring new Loopback interfaces on rbridges that will participate in overlay-gateway, and handling their reachibility through already-configured routing protocols (or static routes) should be fine.
If on other hand we’re setting up a new VCS fabric just to do the HW VTEP, we will need to configure its non-Management IP connectivity (for example, default-vrf) in any case, and then piggy-back VTEP configuration on it. To do so, we could:
- Pick a VLAN ID that will provide L2 connectivity to the overlay-gateway’s VTEP L3 domain. In our example it’s VLAN 150;
- Create an interface “Vlan xxx”, in our case “Vlan 150”, and add it to our VCS fabric’s IP uplink(s), for e.g., Port-channel 20;
- Create a Ve interface, in our case Ve 150, on each rbridge member of our future overlay-gateway. Make sure to have enough IPs available in the subnet. You’ll need one IP per Ve, one for VRRP-E virtual-ip, and one (or more) for your IP uplink, in our case on “Router (Data)”;
- Add VRRP-E configuration to those Ve interfaces, by making them members of the same vrrp-extended-group;
- Make sure the VRF where your Ve interfaces belong (by default – “default-vrf”) has routes to your NSX VTEP subnet(s).
In our case:
interface Vlan 150 name HW_VTEP ! interface Port-channel 20 description Uplink with VLANs for Management (128) and HW VTEP (150) switchport switchport mode trunk switchport trunk allowed vlan add 128,150 spanning-tree shutdown no shutdown ! rbridge-id 101 interface Ve 150 ip proxy-arp ip address 192.168.150.101/24 vrrp-extended-group 150 virtual-mac 02e0.5200.00xx virtual-ip 192.168.150.1 advertisement-interval 1 enable no preempt-mode short-path-forwarding ! no shutdown ! rbridge-id 102 interface Ve 150 ip proxy-arp ip address 192.168.150.102/24 vrrp-extended-group 150 virtual-mac 02e0.5200.00xx virtual-ip 192.168.150.1 advertisement-interval 1 enable no preempt-mode short-path-forwarding ! no shutdown ! rbridge-id 101 ip route 192.168.50.0/24 192.168.150.2 ! rbridge-id 102 ip route 192.168.50.0/24 192.168.150.2
With the above in place, it’s time to do the final check: can our VRRP-E virtual IP reach the VTEPs on ESXi hosts? Let’s check:
VDX6740-101# ping 192.168.50.51 src-addr 192.168.150.1 Type Control-c to abort PING 192.168.50.51 (192.168.50.51) from 192.168.150.1: 56 data bytes --- 192.168.50.51 ping statistics --- 5 packets transmitted, 0 packets received, 100% packet loss
Hmm, no worky, in spite the fact that we’re the VRRP Master:
VDX6740-101# sh vrrp | incl "(VRID |State)" VRID 150 State: Master
So what about the other rbridge, 102? Let’s SSH into it directly, and see:
VDX6740-102# ping 192.168.50.51 src-addr 192.168.150.1 Type Control-c to abort PING 192.168.50.51 (192.168.50.51) from 192.168.150.1: 56 data bytes 64 bytes from 192.168.50.51: icmp_seq=0 ttl=62 time=2.198 ms 64 bytes from 192.168.50.51: icmp_seq=1 ttl=62 time=1.669 ms 64 bytes from 192.168.50.51: icmp_seq=2 ttl=62 time=1.989 ms 64 bytes from 192.168.50.51: icmp_seq=3 ttl=62 time=1.210 ms 64 bytes from 192.168.50.51: icmp_seq=4 ttl=62 time=1.729 ms --- 192.168.50.51 ping statistics --- 5 packets transmitted, 5 packets received, 0% packet loss round-trip min/avg/max/stddev = 1.210/1.759/2.198/0.333 ms
Works fine! What gives?
Our little VCS is connected to upstream network through a Port-channel, which means the return traffic is load-balanced. Looks like in our case the return traffic for this particular conversation ends up on the link that’s connected to the rbridge 102. Never mind that it isn’t VRRP Master; it can still receive and process the traffic, which will be important when it comes to handling VXLAN traffic from ESXi infrastructure.
Ok, given that we can ping an ESXi VTEP just fine, let’s ping in the other direction just for fun before closing off for today:
[root@esx-02a:~] vmkping ++netstack=vxlan 192.168.150.1 PING 192.168.150.1 (192.168.150.1): 56 data bytes 64 bytes from 192.168.150.1: icmp_seq=0 ttl=62 time=9.949 ms 64 bytes from 192.168.150.1: icmp_seq=1 ttl=62 time=2.229 ms 64 bytes from 192.168.150.1: icmp_seq=2 ttl=62 time=2.282 ms --- 192.168.150.1 ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 2.229/4.820/9.949 ms
Works, as expected.
Once we’ve decided on the physical topology for our HW VTEP deployment, it’s time to ensure all the right network connectivity is in place.
HW VTEP requires connectivity to two separate, independent IP domains: Management and VTEP.
Brocade VCS HW VTEP solution uses two different IP interfaces for this connectivity – VCS Virtual IP in Management VRF for connecting to NSX Controllers, and Loopback or VRRP-E based VTEP in Default VRF to talk to ESXi VTEPs. It is possible to set up special VRFs for both of these, if needed.
The choice between the Loopback of VRRP-E comes down to what’s already in place. Where a VCS is already part of a routed domain, Loopback is probably a better solution; while in greenfield deployments where HW VTEP VCS is net-new, using Ve interfaces with VRRP-E might be a better idea.