Surprises from the virtualised networking dept: how to bridge between vCDNI-backed port groups (not)

Summary for the impatient

It appears that vCDNI-backed port groups can’t be bridged together other than via their “uplink” ports. This is because they are not learning bridges (Well, duh!). Enabling promiscuous mode doesn’t help things very much (in fact, it makes them worse), and neither does enabling forged transmits. My conclusion: vCDNI-backed port groups are not an example of “network virtualisation”; however this, as far as I can tell, was done on purpose.

Once upon a time, I was designing standard network deployment scenarios to be used in solution delivery. In addition to vCloud services (self-service and not), we also offer physical servers, logical partitions of physical appliances, and NAS services which people often want integrated with the same network segments as some of their VMs residing on.

We use vCDNI-backed networking with our vCloud Director 1.5, which means we have only one way of getting anything on a vCDNI-backed port group to talk to anything on an outside: via an intermediary VM with one NIC on a vCDNI network segment, and another on a VLAN-backed one. That VM can be a vShield Edge, or some other Virtual Appliance (router, firewall, traffic manager, etc.).

Now, that doesn’t sound as much of a problem, until we realise that the intermediary VM, when configured to do L3 routing, will:

  1. Create another management point to monitor and troubleshoot (itself);
  2. Complicate IP subnetting (with potential effects spreading upstream);
  3. Create a situation when a network segment with VMs has two IP gateways (one default, and one leading to the physical servers/appliances), requiring either additional static routes on each VM, or an unnecessary in-an-out routing on the default gateway device before hitting the real gateway (which may or may not be a big problem, depending on traffic patterns).

 

You might ask, why does it have to be a vCDNI-backed network segment in the first place? Mainly it is because of vShield Edge. We offer an option to use vShield Edge as a self-managed, low-cost perimeter firewall; and the vCD External Routed network that comes with it creates a vCDNI-backed port group for the internal network segment.

Having the above in mind, it was very tempting to see if I could just configure bridging on the intermediary VM and be done with it; however, the attempt has failed miserably. The topology I tried was:

  • “Main” segment #1 (with VMs): an External Routed network
  • “Physical” segment #2 (with physical things on it, in a VLAN): an External Direct network
  • A “virtual appliance” VM, connected to both of these segments (I used Vyatta core 6.5), configured to bridge between its two NICs
  • Two IP hosts (H1 and H2), on the same IP subnet, one connected to the segment #1 and second to the #2
  • Security settings for the segment #1 and #2’s were default – promisc = disable, and MAC changes and forged transmits = enable

 

My observations

  • With the default security settings as above, I could not get IP connectivity going between H1 and H2. ARP requests from H2 do reach H1 because they are sent to the broadcast MAC; but (here starts my speculation) vSwitch won’t send the response back to the non-uplink port where the request has come from, as it isn’t a learning bridge, and doesn’t know that this MAC should be there, because it “shouldn’t” – it’s on a completely separate port group, as far as vSwitch is concerned.
  • Changing security settings on both port groups to promisc = enable did enable connectivity. Somewhat. There were lots of DUPs and lots of lost packets. I didn’t have too much time to investigate, so I don’t know what determines the number of DUPs seen – they appeared to be constant in number, which in my case was 7, for each packet that hasn’t gone missing.

So, a non-starter then. The initially somewhat-elegant solution had to be changed to a less elegant one, which is included below for your “enjoyment” (apologies, it’s a partial cut-n-paste from a more complete diagram, with irrelevant pieces removed). Just imagine that the “VLAN 2” is also connected to the green “Internal XX Management” via another firewall. The blue+light-blue is for SP’s management access to VMs/VAs, and dark-blue+blue is for customer’s internal connectivity. Keep in mind this diagram covers all possible scenarios, and some/many of the components may not be there, depending on a particular design.

Linking vCDNI-backed network segment with a VLAN

Linking a vCDNI-backed network segment with a VLAN

So how does this solution fare against the three complications I listed above? Not too well, I’m afraid:

  1. Yep, still there
  2. Yep, still there
  3. This one is sorted out, as my router VA is a default gateway, which then decides where to send the traffic – to the upstream vSE, to the “physical” segment, or to the customer’s WAN gateway

 

Well, crap. And yes, if we had ToRs that could participate in vCDNI, the life would have been easier. And no, I still don’t like VXLAN; but thanks for thinking about it. 😉

P.S. VMware Edge Gateway should probably take us a good way in to simplify this particular design, when we get there. I think.

About Dmitri Kalintsev

Some dude with a blog and opinions ;) View all posts by Dmitri Kalintsev

5 responses to “Surprises from the virtualised networking dept: how to bridge between vCDNI-backed port groups (not)

  • Ivan Pepelnjak

    The reason for your frustration is the awkward method used to implement vCDNI: it’s a dvFilter module (the technology used to implement NIC-level firewalls). vCDNI is based on Lab Manager product and the company developing that product (before being acquired by VMware) had to use the dvFilter kludge as that was the only network API available to VMware partners at that time.

    Now for the technical details of your problem: a bridging VM would have to run in promiscuous mode (or it wouldn’t receive frames for MAC addresses in the other LAN segment), and the moment you enable promiscuous mode on a VM NIC, it bypasses the dvFilter (no sense to run a firewall on a promiscuous NIC, right?), receiving raw vCDNI-encapsulated packets.

    What can I say – yet another brilliant example of “just good enough” kludge.

    • Dmitri Kalintsev

      Hi Ivan,

      Thanks for the insight. Any ideas around where all the DUPs might be from? They looked absolutely sane on a tcpdump capture – all the right L2 and L3 headers, just… a few too many of them.

      Cheers,

      — Dmitri

  • Tomas Fojta

    This looks very similar to what I was trying to do with VXLAN here:http://fojta.wordpress.com/2012/10/13/poor-mans-vxlan-vlan-bridging/
    I had also some problems with ARP packets but did not investigate it much.

    Did you consider creating VLAN backed network pool for those organization that require bridging? It would be much easier.

    • Dmitri K

      Hi Tomas,

      Thanks for the suggestion – I think that should work; just need to carefully think through the implications of doing that. 🙂

      Cheers,

      — Dmitri

    • Dmitri K

      Tomas,

      We’ve looked into the VLAN-backed network pool option, and it doesn’t appear to be feasible because VLAN ID associated with a vCD network is allocated automatically and there appears to be no option to ensure its persistence.

      — Dmitri

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: