Many of our customers have started using HP BladeSystem c7000 enclosures with ProLiant G6 or G7 blade servers. For the network interconnect they use HP Virtual Connect Flex-10 or FlexFabric blade modules which are ideally connected to the outside world via 10Gb Ethernet network switches. In a less ideal world multiple 1Gb connections can be combined to form a fatter trunk to redundant 1Gb Ethernet core switches.
So much for the cabling! As soon as we dive into the blade enclosure, all network communication stays within the confines of the blade enclosure with its multi-terabit signal backplane.
So how on earth can two virtual machines on two different physical Hyper-V R2 blade servers within this same enclosure only communicate at the speed of only 20 to 30MB per second? Conversely, how can we get them back to a much more acceptable speed? If you want to find out I invite you to read on.
Let me first explain how the different components work together.
In the following diagram we see an HP c7000 Blade Enclosure with several blade servers, two Virtual Connect Flex-10 or FlexFabric blade modules which are each connected to a core Ethernet switch.
A Hyper-V R2 server cannot have enough network adapters. With the latest generation of blade servers we don’t need to fill up the enclosure with a switch pair for each network module or mezzanine. The dual-port 10Gb Ethernet onboard modules can be split into 2 x 4 FlexNICs. Speeds can be dynamically assigned in 100Mb increments from 100Mb to 10Gb.
So the Parent Partition on the Hyper-V R2 server sees 8 network adapters at the speed set in the Virtual Connect server profile. We set up at least three NIC teams for management, virtual machines and cluster communication (heartbeat, live migration, cluster shared volumes). The onboard network adapters in the blade server are from Broadcom and the teaming software used is HP Network Configuration Utility (NCU).
Until last week we used the NCU 10.10.0.x teaming software which allowed us to use native VLANs (see previous blogs). What the older versions have in common is the ultra low speed of VM to VM communication with this particular combination of hardware.
Because we wanted to find out we setup a test configuration with above configuration. The Hyper-V servers were Hyper-V Server 2008 R2 with SP1 (release candidate).
The network performance tests were conducted with NTttcp. This is a multi-threaded, asynchronous application that sends and receives data between two or more endpoints and reports the network performance for the duration of the transfer.
We setup two Windows Server 2008 R2 Virtual Machines with 1 vCPU, 1Gb of memory, 1 Virtual Machine Bus Network Adapter (with IC 6.1.7601.17105) and two VHD’s, one dynamic VHD for the OS and one fixed-sized VHD for the test. No changes were made to the physical or virtual network adapter in terms of TCP and other hardware offloads. We simply kept the defaults.
Test 1: VM to VM on same Hyper-V R2 host in same blade enclosure
| Broadcom driver on host | 5.2.22.0 |
| Teaming software on host | NCU 10.10.0.x |
| Teaming type | Network Fault Tolerance with Preference Order |
| Network speed | 2 x 4Gb (Effectively 4Gb) |
| Total MB Copied | 1.342 |
| Throughput in Mbps | 3.436 |
| Result | Excellent |
| Method | Average of 10 tests |
Test 2: VM to VM on two different Hyper-V R2 hosts in same blade enclosure
| Broadcom driver on host | 5.2.22.0 |
| Teaming software on host | NCU 10.10.0.x |
| Teaming type | Network Fault Tolerance with Preference Order |
| Network speed | 2 x 4Gb (Effectively 4Gb) |
| Total MB Copied | 1.342 |
| Throughput in Mbps | 319 |
| Result | Awful |
| Method | Average of 10 tests |
Via my HP contacts we learnt that HP has been working to improve network performance specifically for HP BladeSystem, Virtual Connect and the FlexNIC network adapters. It turned out that the slow speeds occurred on the LAN on Motherboard (LOM) B, C and D only. So LOM 1:a and LOM 2:a appeared to perform well. If you don’t split your networks into multiple FlexNICs you wouldn’t have noticed any degradation. However, in a Hyper-V cluster environment you need many more networks.
In this iSCSI connected server we use three teams:
-
LOM 1:a + 2:a Management_Team (Domain access; managing Hyper-V hosts)
-
LOM 2:a + 2:b VM_Team (for Virtual Machine communication)
-
LOM 3:c + 3:c Cluster_Team (Live Migration, Cluster Shared Volumes)
The remaining two ports are used to create a MPIO bundle for connection to the iSCSI network.
Because the VM_Team is on LOM B, we suffered very low performance when two VM’s living on different Hyper-V hosts.
To see if the newly built drivers and teaming software (released on december 19th) we updated the Broadcom drivers and the NIC teaming software. The same tests were executed to see the difference.
Test 1: VM to VM on same Hyper-V R2 host in same blade enclosure
| Broadcom driver on host | 6.0.60.0 |
| Teaming software on host | NCU 10.20.0.x |
| Teaming type | Network Fault Tolerance with Preference Order |
| Network speed | 2 x 4Gb (Effectively 4Gb) |
| Total MB Copied | 1.342 |
| Throughput in Mbps | 4.139 (+21,7%) |
| Result | Excellent |
| Method | Average of 10 tests |
Test 2: VM to VM on two different Hyper-V R2 hosts in same blade enclosure
| Broadcom driver on host | 6.0.60.0 |
| Teaming software on host | NCU 10.20.0.x |
| Teaming type | Network Fault Tolerance with Preference Order |
| Network speed | 2 x 4Gb (Effectively 4Gb) |
| Total MB Copied | 1.342 |
| Throughput in Mbps | 1363 (+426%) |
| Result | Good |
| Method | Average of 10 tests |
Although we haven’t tested and compared all LOMs we feel quite confident that network bandwidth is now distributed in a more even way across the different FlexNICs.
©2010 Hans Vredevoort