WoodITWork.com

It's about time I let the world know what I was thinking...

Flex-10 ESX design with simplicity and scalability: Part 1

By Julian Wood
bladesesx-2flex-10hp-2networkingvmware-2

I’ve written quite a few posts about HP Flex-10 and some of the challenges and solutions to getting everything up and running.

I’ve also discussed my ideas about Flex-10 ESX design on the vSoup.net podcast so here it is…

If you are deploying Flex-10 make sure you have all the prerequisites in place: /2010/08/09/flex-10-esx-pre-requisites/

I also recently managed to find the manual page for the HP Virtual Connect Flex-10 10Gb Ethernet Module for c-Class BladeSystem on HP’s site which is a good reference launch page for the latest HP Virtual Connect Ethernet Cookbook and all other Flex-10 related documentation. Don’t you love trying to find things on HP’s site? http://h20000.www2.hp.com/bizsupport/TechSupport/DocumentIndex.jsp?lang=en&cc=us&contentType=SupportManual&prodTypeId=3709945&prodSeriesId=3794423&docIndexId=64180

I do however think that HP is trying a little too hard to sell all the benefits of Flex-10 and is possibly sacrificing simplicity to show off all the features of Flex-10. They seem to want you to cram us much of Flex-10 into your deployment as possible when you should rather be streamlining the design to rather give you only what you need.

One of the goals of this blog is simplifying IT so it’s time to apply this to Flex-10.

Stacking

Let’s start with how to get your chassis talking to each other.

HP c7000 Chassis’ with Flex10 switches are meant to be joined together. You can join 4 chassis together but its more common to join 3 together as it fits better in your rack. It may be a good idea to name your chassis from the bottom of the rack up starting with Chassis A because if you are starting with 2 x chassis you will have Chassis A and Chassis B and later can add another Chassis C to fill up your rack. If you have A at the top with 2 chassis, when you add Chassis C the order will be confusing.

You will have 1Gb Onboard Administrator network cables from each OA module to an upstream switch providing chassis management and iLO for your blades.

You will then cascade the OA modules together using ethernet network cables so you can manage all your chassis by connected to any one of them.

That takes care of the cabling for chassis administration.

You will then hopefully have purchased 2 x Flex-10 switches for each chassis and inserted them in Bay 1 and 2. What you want to do is link all these Flex-10 switches together with HP stack cables so they form a single logical network and traffic from blades in one chassis can travel to switches or blades in another chassis without having to go to an upstream switch.

Adjacent switches in a chassis are linked together via the chassis backplain. You then need to create a ring to connect all switches together and provide 2 directions for traffic to pass. Connect 10Gb CX-4 HP Stacking cables to the X1 connector to link the chassis together.

This is what your stacking cabling will look like. Orange lines are internal stacking links and red lines are external cables. For a 3 x chassis deployment you will need 3 x CX-4 stack cables.

If your stacking is all correct within Virtual Connect Manager you will see the following:

You can get more chassis stacking information from HP’s Virtual Connect Multi-Enclosure Stacking Reference Guide.

Cooking with Flex-10

HP’s Virtual Connect Ethernet Cookbook:Single and Multi Enclosure Domain (Stacked) Scenarios is a good start to see what is possible with Flex-10 networking.

Although it is a comprehensive document at 229 pages, I don’t think it does a good job of helping you decide which scenario you should deploy. It does give many scenarios but without discussing the merits of each.

As I said in the beginning, I get the impression HP is trying just a little too hard to show how great Flex-10 is and sacrificing simplicity in the process so you go through the whole document and are still none the wiser as to which scenario you should go for.

The design I recommend isn’t part of the cookbook in its entirety but is actually there in parts spread across multiple scenarios.

What are you trying to achieve with your ESX networking and Flex-10?

For this design I would like to achieve the following:

  1. Use BL460 or BL490 blades without needing additional mezzanine cards
  2. Provide fault tolerant network connectivity to all ESX hosts
  3. Use NFS or iSCSI for storage traffic so its all ethernet and you don’t have to pay extra for fiber channel
  4. Have enough bandwidth available for VM Traffic, Storage Traffic, Management Traffic and vMotion traffic
  5. Segregate VM and Storage traffic so during normal operation they don’t share bandwidth
  6. Separate vMotion traffic so it doesn’t compete with VM/Storage traffic and contain the traffic within a rack keeping it more secure
  7. Support multiple VLANs for VM traffic
  8. Use all available network uplinks so you don’t waste 10GbE ports being idle or standby
  9. Allow easy future expansion capacity for additional networking
  10. Keep it simple!

Flex-10 technology allows you to partition each of the 10GbE Nics on a blade into 4 x Nics and partition the 10GbE between the 4 x Flex-Nics, which is where the Flex(ible) part comes in. This means a BL460/490 blade with 2 onboard 10GbE Nics can see 8 x Nics. Although these Nics are logical from the Flex-10 point of view the blade sees them as 8 separate physical Nics with 8 different MAC addresses.

If you were to install Windows on a blade you would see the following devices:

If you were to install ESX you would see the following:

Each Flex-Nic is named individually in Virtual Connect with the LOM (Lan on Motherboard) identifier and corresponds to an ESX vmnic number which will be mapped later to a Virtual Connect Ethernet Network.

[table id=3 /]

Each of these 8 x Nics is then assigned to an ethernet network that is created within Virtual Connect Manager. Some of these networks may in turn have uplinks assigned so the Flex-Nic can talk to the LAN or they may be internal to the rack. You can think of these 8 x Flex-Nics as 8 x Nics coming out of a rack mounted server and you can choose which ones get connected to upstream switches. This is what the Flex-10 logical layout looks like.

Breaking down some of the design goals we can start to see what traffic we need to support. Service Console traffic is very minimal so could be shared with VM and/or storage traffic. We need a Nic for VM traffic and we need a Nic for storage traffic. We need a separate Nic for vMotion traffic as we don’t want to risk flooding VM/storage traffic with vMotion traffic.

Looking at VM and storage traffic we need to ensure we have redundency for the network traffic. As we have a share of 10GbE available per Flex-Nic we could however create a team of 2 x Flex-Nics to provide redundency and use ESX port groups to redirect VM traffic over 1 x Flex-Nic and storage traffic over 1 x Flex-Nic but be able to use the other Nic in the pair for redundency.

We also need to provide redundency for vMotion traffic as well so that would be another 2 x Flex-Nics.

So, with 4 x Flex-Nics we can satisfy all traffic requirements.

HP’s cookbook scenarios try and use as many of these 8 x Flex-Nics as possible to show off Flex-10 but I prefer to use 4 x Flex-Nics so you have spare Nics available if in the future you need to be able to add functionality to additional networks.

The first pair of Nics, 1A and 2A, carrying VM and Storage traffic need to talk outside the rack to upstream switches. The second pair of Nics, 1B and 2B, carrying vMotion traffic don’t need to talk outside the rack as vMotion traffic doesn’t need to appear on the LAN and as it is not encrypted we also want to keep it separate from LAN traffic.

From an individual blade perspective this is what the networking will look like:

VLANs

Blades are all about a converged infrastructure. You purchase blades so you can share power and networking and make it quicker to be able to provision servers. You still however want to be able to segregate traffic and this is where port groups and also VLANs come in.

There are two operational modes for HP Virtual Connect. Mapped Mode and Tunneled Mode. Mapped Mode allow you to create Ethernet Networks and manage VLANs within Virtual Connect. Tunneled VLANs allow you to create Ethernet Network but pass through all traffic through Virtual Connect whether they are VLAN tagged or not. As ESX has very good support for VLANs this design uses Tunneled mode to pass all VLANs through the Virtual Connect switches and uses ESX networking to manage the VLANs using Port Groups.

Using Tunneled Mode also means you only need to manage your VLANs at the upstream switch. Any VLANs created on the upstream switches and trunked down the uplinks will be passed through Virtual Connect and be directly available to ESX.

This means you don’t have to create and manage the VLANs on your upstream switches as well as in your Virtual Connect Domain.

VLANs are a great way to manage network capacity. It’s a good idea to create multiple VLANs to separate different traffic. Create a VLAN for your physical host IP addresses (Service Console/Management Network and vmkernel) and multiple VLANs for your VM traffic. Plan ahead for your VM capacity and ensure you have enough IP addresses to grow into. If your VDI environment could grow into 2000 VMs then why not even double your requirement for your VLAN planning just in case you buy another company or combine two datacenters and then you don’t have to worry about adding capacity later.

So, how can we make all this happen with Virtual Connect?

First you will need to have created your Virtual Connect domain and imported all your chassis. The instructions for this are in the HP Cookbook so I won’t repeat here.

Then you need to Set the Ethernet Settings | Advanced Settings and select Tunnel VLAN Tags.

Flex-10 Networking

Next you want to create your Ethernet Networks. Create a separate Ethernet Network for each Flex-Nic so that would be 8. The reason for this is that you to be able to manage the networking for each Flex-Nic separately. Network redundency will be a function of ESX to handle failover so you don’t need to have multiple Flex-Nics connected to a single Ethernet Network to achieve this.

[table id=4 /]

When you have created your Ethernet Networks in Virtual Connect Manager they should look like this:

This is normally the time to add uplinks to your Ethernet Networks but hold off for now as its worth explaining the link between Ethernet Networks and server profiles.

You need to connect each Flex-Nic to its Ethernet Networks which is done as part of the server profile.

Create the server profiles for each blade creating 8 x Networks Connections and then mapping the Flex-Nic LOM to the Ethernet Network Name.

Here is where you can allocate Bandwidth if you need to. I’ve just split the 10GbE evenly between the 4 x Flex-Nics. This would mean you have 2.5 Gb available for VM LAN traffic, another 2.5 Gb available for vmkernel storage traffic and 2 x 2.5 Gb available for vMotion.

I think this is plently of bandwidth. Remember your network bottleneck is unlikely to be your blade as you will be sharing 10GbE uplinks to your upstream switches between all the blades in your rack. Your NFS/iSCSI NAS Server probably has 10GbE available bandwidth and you are connecting 48 blades to this so 2.5 Gb per blade is plenty.

The same server profiles need to be created for all blades.

Now we will add the uplinks. The Ethernet Networks for LAN and NAS traffic need to be connected to the external world using a design goal to use all uplinks for traffic and not have passive 10GbE connections doing nothing.

The 6 x Flex-10 switches are acting as a single logical stacked switch. Each blade is connected to the same set of Ethernet Networks so traffic on for example the vm_trunk_1a network is passing through each Flex-10 switch through the stacking cables. This means you can connect the vm_trunk_1a Ethernet Network to uplinks from any switch in the stack.

This stacking is what allows vMotion to be contained within the rack. Every blade can see every other blade in the rack over the stacking links using the vm_vmotion_1b and vm_vmotion_2b networks without having to have any uplinks assigned.

As we are going to be using ESX port groups and ESX failover order settings to direct VM traffic and NAS traffic over separate Nics, think of your rack as being split in two vertically with LAN traffic travelling over the left hand side and NAS traffic over the right hand side.

Let’s look at the bandwidth and cabling options that are possible.

20Gb Option

The simplest design would be having 2 x 10GbE uplinks from your rack going to upstream switches. This will provide you with 20Gb total bandwidth for both LAN and NAS.

  • Run a cable from the Flex-10 Switch in Chassis A Bay 1 Port X2 up to a 10GbE port on Upstream Switch 1. Through this all blade LAN traffic will be primarily directed.
  • Run a cable from the Flex-10 Switch in Chassis C Bay 2 Port X2 up to a 10GbE port on Upstream Switch 2. Through this all blade NAS traffic will be primarily directed.

Configure the upstream switch ports as trunked access ports and add all vlans you will require for your ESX hosts.

Within Virtual Connect Manager:

  • Edit the vm_trunk_1a Ethernet Network and add the Uplink in Chassis A Bay 1 Port X2
  • Edit the vm_trunk_2a Ethernet Network and add the Uplink in Chassis C Bay 2 Port X2

Once you have added the External Uplink Ports to your Ethernet Network ensure you enable Smart Link and Enable VLAN Tunneling.

Smart Link is the HP technology that tells an individual Flex-Nic its associated uplinks are down so a blade can use its own network teaming software to fail over network traffic. As the Flex-10 Nics are hard wired in the chassis to the Flex-10 switches they do not go down so you need something to be able to tell the Flex-10 adapter that it is not connected to the external network. Smart Link is what allows this down to an individual Flex-Nic.

Your Ethernet Networks should now look like this:

40Gb Option

If 20Gb of bandwidth will not support your needs you can easily double the bandwidth to 40GbE by running a second cable for each pair.

  • Run a cable from the Flex-10 Switch in Chassis A Bay 1 Port X2 up to a 10GbE port on Upstream Switch 1. Through this all blade LAN traffic will be primarily directed.
  • Run a cable from the Flex-10 Switch in Chassis A Bay 1 Port X3 up to a 10GbE port on Upstream Switch 1. Through this all blade LAN traffic will be primarily directed.
  • Run a cable from the Flex-10 Switch in Chassis C Bay 2 Port X2 up to a 10GbE port on Upstream Switch 2. Through this all blade NAS traffic will be primarily directed.
  • Run a cable from the Flex-10 Switch in Chassis C Bay 2 Port X3 up to a 10GbE port on Upstream Switch 2. Through this all blade NAS traffic will be primarily directed.

You will need to create a LACP group on each of your upstream switches and put both uplink ports into the group.

Configure the upstream switch ports within the LACP groups as trunked access ports and add all vlans you will require for your ESX hosts.

This will allow all ports to be active so you will have 40Gb available.

Within Virtual Connect Manager:

  • Edit the vm_trunk_1a Ethernet Network and add the Uplinks in Chassis A Bay 1 Port X2 and Port X3
  • Edit the vm_trunk_2a Ethernet Network and add the Uplinks in Chassis C Bay 2 Port X2 and Port X3

Once you have added the External Uplink Ports to your Ethernet Network ensure you enable Smart Link and Enable VLAN Tunneling.

If your LACP groups are set up correctly, you should see all uplinks as Linked-Active.

Your Ethernet Networks should now look like this:

40Gb vPC Option

If you are using a Cisco Nexus series switch you can improve failover by splitting your LACP groups across the upstream switches by using a VPC. For a LACP group to be formed both uplinks need to be in the same Flex-10 Switch but with a VPC this can be in separate upstream switches.

  • Run a cable from the Flex-10 Switch in Chassis A Bay 1 Port X2 up to a 10GbE port on Upstream Switch 1. Through this all blade LAN traffic will be primarily directed.
  • Run a cable from the Flex-10 Switch in Chassis A Bay 1 Port X3 up to a 10GbE port on Upstream Switch 2. Through this all blade LAN traffic will be primarily directed.
  • Run a cable from the Flex-10 Switch in Chassis C Bay 2 Port X2 up to a 10GbE port on Upstream Switch 1. Through this all blade NAS traffic will be primarily directed.
  • Run a cable from the Flex-10 Switch in Chassis C Bay 2 Port X3 up to a 10GbE port on Upstream Switch 2. Through this all blade NAS traffic will be primarily directed.

You will need to create VPC groups for each uplink pair across both your upstream switches and put both uplink ports into the group.

Configure the upstream switch ports within the VPC groups as trunked access ports and add all vlans you will require for your ESX hosts.

Once you have added the External Uplink Ports to your Ethernet Network ensure you enable Smart Link and Enable VLAN Tunneling.

Your Ethernet Networks will be configured in the same way as the 40Gb Option.

If your VPCs are set up correctly, you should see all uplinks as Linked-Active.

In Part 2 we’ll continue and look at how the Flex-10 networking is presented and configured in ESX.