Serving bandwidth-hungry VMs with DC fabrics and NSX for vSphere

It is fairly common for VMs to talk to SMB / NFS shares hosted by physical storage arrays, or send their backups to physical appliances.

In this blog post, we’ll have a look at connectivity options for this use case.

When such VMs are connected to hypervisor-based virtualised networks (for example, NSX Logical Switches), all their traffic to physical endpoints would normally go either through a layer of Edge Services Gateways (ESG), or through an NSX software-based VXLAN to VLAN bridge.

Note: to use NSX software bridging, Logical Switches must be contained to a single DVS, and VLAN being bridged must be added to Uplinks of all ESXi hosts in that DVS.

Shared datapath
Shared datapath

To recap, traffic from VM to the appliance flows from the VM to its local DLR, then via VXLAN over IP to the Edge host where it is routed by an ESG to an Uplink VLAN, that has an L3 gateway in a physical fabric. Physical router in fabric then routes it to the Storage / Backup VLAN, where it reaches the appliance.

Return traffic flows from the appliance back to ESG, which then sends it to its local DLR, where it is routed to the VM’s Logical Switch, and forwarded with VXLAN encapsulation back to VM’s host.

Application traffic to/from users follows the same path.

In many cases volume of storage / backup traffic is reasonably low. It is then perfectly fine for that traffic to follow the same path as general application network data.

In other cases it may be better to separate these flows. There could be a number of reasons for doing that:

  • You want to ensure that customers’ connections to applications are never affected by bursts in backend communications
  • You need to maximise the performance between application and file shares through unconstrained bandwidth and minimised latency
  • You have to separate user and backend traffic for security reasons, for example if you are a Service Provider offering managed backup service to tenants’ VMs.

So how can we solve this?

We can meet our needs by creating a special purpose L2 network segment connected to the shared physical appliance (storage array or backup system), and making it available to the VMs that need it via an additional dedicated virtual NIC (vNIC).

There are two ways you can implement this L2 network segment:

  • As a stretched storage / backup VLAN; or
  • As a storage / backup Logical Switch bridged to a matching VLAN.

Let’s have a look at both in more detail.

Solution #1: Dedicated VLAN presented to all ESXi hosts

In the simplest case, you would create a VLAN for your storage array / backup server, add it to the Uplink interfaces of your ESXi hosts, and create a matching VLAN-backed dvPortgroup. Then you add a new vNIC to each VM that needs access to the file shares or backup server, and connect it to that new dvPortgroup. One VLAN, one IP subnet.

Last step is to create an NSX DFW rule applied to everything connected to that dvPortgroup, with Source = this dvPg, Destination = this dvPg, Protocol = Any, Action = Deny. This will prevent the VMs potentially belonging to different applications / tenants or business units from reaching each other.

For this approach to work, we must be able to stretch the storage / backup VLAN to all our ESXi servers that may be located in different racks.

Stretched VLAN
Stretched VLAN

VMs from different Apps and/or security zones connect to the same shared storage / backup VLAN.

In addition to an old-school DC network with VLANs / vPCs / MLAGs / Spanning Tree, there are two types of modern DC fabric that can support this solution while being more scalable, stable, and easier to configure and operate:

  1. A self-forming L2 fabric, such as Brocade VCS; or
  2. A self-provisioning L3 fabric with BGP EVPN.

There are no real scalability limits here apart from the number of VMs that you’re willing to connect to that shared VLAN. A /22 is probably is as large as I would be comfortable with from “many eggs in one basket” point of view. To scale further, just create another VLAN with another /22 subnet. This assumes that your storage array / backup server can support multiple VLAN interfaces. You’ll need to connect it to each of these shared VLANs, with an IP from each matching subnet.

The operational overhead of this solution is low – the shared VLAN is added to ToRs and ESXi hosts’ Uplinks once, dvPortgroup is also created once. There is no need to set up route tables on VMs since the storage array / backup server is on the same subnet as the VM’s dedicated vNIC.

Once set-up is in place, all high bandwidth conversations between VMs and storage array / backup server happen over this dedicated vNIC / dvPortgroup / VLAN.

Since this is a dedicated dvPortgroup, you can also configure its Teaming settings so that it uses different Uplink(s) from those of your VXLAN Logical Switches.

This solution is also appropriate in cases where you have older physical NICs in your ESXi servers that have less than stellar performance when handing VXLAN.

As mentioned above, this works great for a DC network or fabric that can provide end to end VLANs. However if you have an L3 fabric with VLANs localised to individual racks, we won’t be able to stretch the VLAN.

Assuming your L3 fabric doesn’t support BGP EVPN, a different approach is needed. In this case, we need to turn to..

Solution #2: Dedicated Logical Switch bridged to a VLAN through a HW VTEP

NSX Logical Switches are designed to extend L2 over L3 networks. They also require no changes to your ESXi hosts’ Uplink configuration – no VLANs to add or remove when things change.

Therefore instead of using a dedicated storage / backup VLAN-backed dvPg, we can create a Logical Switch, and then use a Hardware VXLAN Gateway (HW VTEP) to bridge it with a storage / backup VLAN. Since one of the defining features of HW VTEPs is how scalable and performant they are, we should be in good shape here.

The rest of the setup is exactly as before:

  • We create an NSX DFW rule to prohibit VMs on our new LS from talking to each other;
  • We add a new “storage / backup” vNIC to our bandwidth-hungry VMs;
  • We connect it to our new LS bridged to the storage / backup VLAN, and give it an IP address from our /22; and
  • Congratulate ourselves on making the right life choices, call it a day, and go out for our favourite beverage.
Hardware VTEP
Hardware VTEP

Compared to the Solution 1, there’s one thing to keep in mind – all high bandwidth conversations with physical appliances will happen through VXLAN VTEPs on ESXi hosts. It is thus important to consider these additional bandwidth demands when designing NSX uplinks. VMware-recommended multi-VTEP configuration with Source Port load balancing should work perfectly fine since you have many source VMs, allowing you to get good load spread across pNICs.


There are configurations where it makes sense to separate network path to shared bandwidth-hungry physical infrastructure, such as file shares or backup servers, from front-end application data.

NSX Distributed Firewall, in combination with L2 fabric or a Hardware VTEP, enables you to do this with a low set-up and on-going overheads, as described above.

Hope you found this useful!

Next post in the series is here.

About Dmitri Kalintsev

Some dude with a blog and opinions ;) View all posts by Dmitri Kalintsev

3 responses to “Serving bandwidth-hungry VMs with DC fabrics and NSX for vSphere

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: