Summary for the impatient
It appears that vCDNI-backed port groups can’t be bridged together other than via their “uplink” ports. This is because they are not learning bridges (Well, duh!). Enabling promiscuous mode doesn’t help things very much (in fact, it makes them worse), and neither does enabling forged transmits. My conclusion: vCDNI-backed port groups are not an example of “network virtualisation”; however this, as far as I can tell, was done on purpose.
Once upon a time, I was designing standard network deployment scenarios to be used in solution delivery. In addition to vCloud services (self-service and not), we also offer physical servers, logical partitions of physical appliances, and NAS services which people often want integrated with the same network segments as some of their VMs residing on.
We use vCDNI-backed networking with our vCloud Director 1.5, which means we have only one way of getting anything on a vCDNI-backed port group to talk to anything on an outside: via an intermediary VM with one NIC on a vCDNI network segment, and another on a VLAN-backed one. That VM can be a vShield Edge, or some other Virtual Appliance (router, firewall, traffic manager, etc.).
Now, that doesn’t sound as much of a problem, until we realise that the intermediary VM, when configured to do L3 routing, will:
- Create another management point to monitor and troubleshoot (itself);
- Complicate IP subnetting (with potential effects spreading upstream);
- Create a situation when a network segment with VMs has two IP gateways (one default, and one leading to the physical servers/appliances), requiring either additional static routes on each VM, or an unnecessary in-an-out routing on the default gateway device before hitting the real gateway (which may or may not be a big problem, depending on traffic patterns).
You might ask, why does it have to be a vCDNI-backed network segment in the first place? Mainly it is because of vShield Edge. We offer an option to use vShield Edge as a self-managed, low-cost perimeter firewall; and the vCD External Routed network that comes with it creates a vCDNI-backed port group for the internal network segment.
Having the above in mind, it was very tempting to see if I could just configure bridging on the intermediary VM and be done with it; however, the attempt has failed miserably. The topology I tried was:
- “Main” segment #1 (with VMs): an External Routed network
- “Physical” segment #2 (with physical things on it, in a VLAN): an External Direct network
- A “virtual appliance” VM, connected to both of these segments (I used Vyatta core 6.5), configured to bridge between its two NICs
- Two IP hosts (H1 and H2), on the same IP subnet, one connected to the segment #1 and second to the #2
- Security settings for the segment #1 and #2’s were default – promisc = disable, and MAC changes and forged transmits = enable
- With the default security settings as above, I could not get IP connectivity going between H1 and H2. ARP requests from H2 do reach H1 because they are sent to the broadcast MAC; but (here starts my speculation) vSwitch won’t send the response back to the non-uplink port where the request has come from, as it isn’t a learning bridge, and doesn’t know that this MAC should be there, because it “shouldn’t” – it’s on a completely separate port group, as far as vSwitch is concerned.
- Changing security settings on both port groups to promisc = enable did enable connectivity. Somewhat. There were lots of DUPs and lots of lost packets. I didn’t have too much time to investigate, so I don’t know what determines the number of DUPs seen – they appeared to be constant in number, which in my case was 7, for each packet that hasn’t gone missing.
So, a non-starter then. The initially somewhat-elegant solution had to be changed to a less elegant one, which is included below for your “enjoyment” (apologies, it’s a partial cut-n-paste from a more complete diagram, with irrelevant pieces removed). Just imagine that the “VLAN 2” is also connected to the green “Internal XX Management” via another firewall. The blue+light-blue is for SP’s management access to VMs/VAs, and dark-blue+blue is for customer’s internal connectivity. Keep in mind this diagram covers all possible scenarios, and some/many of the components may not be there, depending on a particular design.
So how does this solution fare against the three complications I listed above? Not too well, I’m afraid:
- Yep, still there
- Yep, still there
- This one is sorted out, as my router VA is a default gateway, which then decides where to send the traffic – to the upstream vSE, to the “physical” segment, or to the customer’s WAN gateway
Well, crap. And yes, if we had ToRs that could participate in vCDNI, the life would have been easier. And no, I still don’t like VXLAN; but thanks for thinking about it. 😉
P.S. VMware Edge Gateway should probably take us a good way in to simplify this particular design, when we get there. I think.