WoodITWork.com

I’ve written quite a few posts about HP Flex-10 and some of the challenges and solutions to getting everything up and running.

I’ve also discussed my ideas about Flex-10 ESX design on the vSoup.net podcast so here it is…

If you are deploying Flex-10 make sure you have all the prerequisites in place:
http://www.wooditwork.com/2010/08/09/flex-10-esx-pre-requisites/

I also recently managed to find the manual page for the HP Virtual Connect Flex-10 10Gb Ethernet Module for c-Class BladeSystem on HP’s site which is a good reference launch page for the latest HP Virtual Connect Ethernet Cookbook and all other Flex-10 related documentation. Don’t you love trying to find things on HP’s site?
http://h20000.www2.hp.com/bizsupport/TechSupport/DocumentIndex.jsp?lang=en&cc=us&contentType=SupportManual&prodTypeId=3709945&prodSeriesId=3794423&docIndexId=64180

I do however think that HP is trying a little too hard to sell all the benefits of Flex-10 and is possibly sacrificing simplicity to show off all the features of Flex-10. They seem to want you to cram us much of Flex-10 into your deployment as possible when you should rather be streamlining the design to rather give you only what you need.

Read more…

Sometimes you need to delve into the innards of ESX to get information out that just isn’t available through the VMware SDK and so not directly available to PowerCLI.

If you’ve been doing any work recently with HP Flex-10 and its firmware requirements, you may need to find out the network card firmware version which is only available from within the ESX console by typing ethtool -i vmnic0

So, PowerCLI to the rescue, but actually using PowerCLI to call Plink.exe which is Putty’s command-line connection tool. You can use this same process to pull anything that you can get from the ESX console as long as you can parse the results of the command and find the information you need.

Read more…

I was honoured to be invited as the first guest on vSoup.net joining Chris Dearden (@chrisdearden), Ed Czerwin (@eczerwin) and Christian Mohn (@h0bbel) talking about virtualisation

We spoke about managing virtual desktops, VDI, HP Flex-10 switches and firmware issues, storage and plently of other things!

You can download the podcast from vsoup.net or even better subscribe through iTunes.

I’ve had confirmation from HP that the ESX 4.1 updated Broadcom 1.60 driver DOES support DCC/SmartLink.

I had queried this in my previous post HP Flex-10, ESX and Broadcom fun continues…

So, looks like the 1.60 driver is the one to go for on ESX 4.1 as long as you have the Flex-10 firmware prerequisites and update the Heap Memory allocated by the Broadcom bnx2x driver as listed here:
http://kb.vmware.com/kb/1031805

Unfortunately this driver isn’t available through Update Manager but follow my previous post and you can use PowerCLI.

I’ve also updated my Flex-10 ESX pre-requisites post so you have all the information in one place.

HP also told me they plan to update their original advisory and get better communication on the VMware site to mention DCC/SmartLink, let’s hope this happens.

I’ve been updating my HP Flex-10 firmware information to reflect what’s been happening with ESX 4.1 and it looks like the fun is still ongoing.

My Flex-10 ESX pre-requisites post has also been updated with the latest information.

ESX 4.1 was released with an inbox Broadcom bnx2x driver 1.54.1.v41.1-1vmw

This driver was found to cause Network connectivity issues and PSODs which I warned about and at that stage recommended not deploying ESX 4.1 if you had affected Nics.

VMWare’s KB article: http://kb.vmware.com/kb/1029368 – VMware ESX/ESXi 4.1 host with Broadcom bnx2x (in-box driver version 1.54.1.v41.1-1vmw) experiences a loss of network connectivity and a purple diagnostic screen.

This driver also unfortunately took a few steps back by removing DCC/SmartLink which I mentioned in this post.

Read more…

VMware has released a new Broadcom bnx2x driver today and made it available through Update Manager.
http://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&externalId=1029394

The update looks like it fixes the critical issues referenced in my previous post.

What is very strange though is the 4.1 bundled version number was:
1.54.1.v41.1-1vmw
The new version number is:
1.54.1.v41.1-2vmw

See the difference, a small little 2. So this looks like a very minor release. What is the 1.60 driver all about then?

Does this minor 1.54-2 release fix the issues with the built in driver but not fix SmartLink / DDC. Does the 1.60 driver fix the issues with the built-in driver or just fix SmartLink / DDC?

I haven’t been able to find any more information on what the 1.60 driver fixes or breaks as there doesn’t seem to be any published information about it.

I’m very happy these drivers are now being released through Update Manager which is the easiest deployment method although you can still use PowerCLI to deploy them but why release 1.60 and then release 1.54-2 later?

I’ve said it before: Come on everyone…VMware, Broadcom, HP need to get together and sort this out!

After my previous post about vCenter availability I thought I should expand on some other factors related to availability and what you should be thinking about to protect your business against failure.

Too often IT solutions are put in place without properly considering what could go wrong and then people get suprised when they do. Sometimes the smallest things can make the biggest difference and you can land up with your business not operating because some very small technical glitch that could have been avoided brings everything down.

Planning for failure should be a major part of any IT project and the cliche is certainly true, “If you fail to plan you plan to fail.” Planning for failure includes big things like a full site disaster recovery plan but also includes small things like ensuring all your infrastructure components are redundent and you don’t have any single points of failure.

Read more…

I’ve managed to find a good white paper document on HP’s site which may help Cisco people understand more about Virtual Connect and Flex-10.

http://h20000.www2.hp.com/bc/docs/support/SupportManual/c01386629/c01386629.pdf

There’s good information to explain what often freaks out Cisco people about Virtual Connect, no requirement for spanning tree.  Also included is the explanation of Flex-Nics, switch stacking and how best to configure the Cisco side of the connection.

The blub according to HP:

“A technical discussion of the HP Virtual Connect 3.1x features and use with a Cisco Catalyst network infrastructure”

Updated: 28 October 2013 to include new HP September bundles and release of ESXi 5.5.

Many companies are currently going through major infrastructure changes as the reach of virtualisation into IT increases and they start moving towards a cloud model.

Blade server use is growing as people start to think more seriously about large server farms and building the infrastructure for a “cloud”, be it internal(private) or external (public). HP has been doing a lot of work selling its blades (BL460C and BL490C) in C7000 chassis with Flex-10 switches.

These infrastructure changes can be pretty complicated to implement as you need to plan for a whole new server hosting environment, not just adding another server into a rack somewhere.

In a converged world you also need to get server people, power people, VM people, storage people and networking people to work together which is a challenge at the best of times in traditional siloed enterprise IT departments.

HP has however had some problems over the years delivering the technical message to get this all done.  Unfortunately there often wasn’t enough clear information from HP about what needs to be done. Their reference architectures and Flex-10 cookbook were incomplete or out of date and in some cases contradictory and misleading. This has improved somewhat over the years but it doesn’t help that HP’s website must rate as one of the worst to find anything and it’s an IT company…quite a big one as well and it lets itself down when its hardware in my opinion is fantastic.

So, here’s a pre-requisite list I’ve been building up and maintaining over the years to hopefully get you on your way.

Onboard Administrator Firmware

HP BladeSystem c-Class Onboard Administrator (OA) Firmware

Update all your chassis OA firmware to 4.01. Although this is a v4 release the update is not nearly as dramatic as Virtual Connect 4. The Onboard Administrator upgrade from 3.71 to 4.01 just adds IPv6 support, new hardware models and a raft of bug fixes.

Update all your chassis OA firmware to 3.71.

http://h20566.www2.hp.com/portal/site/hpsc/template.PAGE/public/psi/swdDetails/?sp4ts.oid=3692952&spf_p.tpst=swdMain&spf_p.prp_swdMain=wsrp-navigationalState%3Didx%253D0%257CswItem%253DMTX_b9dd1a1e38b8457498123b9c52%257CswEnvOID%253D4064%257CitemLocale%253D%257CswLang%253D%257Cmode%253D3%257Caction%253DdriverDocument&javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken

Older version information:

There is a problem with Emulex firmware prior to version 4.1.450.7 that can result in SmartLink otherwise known as Device Control Channel (DCC) not working with a 10Gb physical link when you have full height blades. When you upgrade the OA, you may lose network connectivity. Update the NIC firmware and look at Customer Advisory c03600027 before upgrading the OA.

If you are using Gen 8 blades you will have to update your OA firmware to at least 3.50 which is the latest Recommended release as although the blades will be recognised there will be issues with Virtual Connect.

A newer version 3.55 was released the next day which fixed some authentication and EBIPA issues which must have slipped through the testing net with 3.50 (great!) . OA 3.50 can be downloaded from: http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=3709945&prodSeriesId=1844065&prodNameId=3692952&swEnvOID=4064&swLang=8&taskId=135&swItem=MTX-00b278b5e9194e7ab0ab362d55&mode=3

OA 3.55 can be downloaded from:

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=3709945&prodSeriesId=1844065&prodNameId=3692952&swEnvOID=4064&swLang=8&taskId=135&swItem=MTX-b22a6014742b4bd7a0a05557b9&mode=3

Virtual Connect Firmware

HP BladeSystem c-Class Virtual Connect Ethernet (VC-Enet), VC 4Gb FC, VC 8Gb 20-port FC, and VC 8Gb 24-port FC modules

Update your Virtual Connect Flex-10 switches to firmware 3.75 as this is a Recommended release with many bug fixes.

Virtual Connect 4.01 has been released which includes many more features and has also since been updated to 4.1 . It’s still relatively new so if you are going to use it, do your testing.

Upgrade Alert: If you have both single dense and double dense servers in your chassis and you are upgrading from Virtual Connect firmwares 3.10-3.60 up to Virtual Connect firmwares 3.70-3.75 you need to be aware of a serious issue where upgrading the firmware may cause your server profiles to be unassigned which will cause a network outage.

See this post: HP Virtual Connect Firmware upgrade may cause Server Profiles to be unassigned causing a network outage

VC 3.75 can be downloaded from: http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=3709945&prodSeriesId=3794423&prodNameId=3794431&swEnvOID=4091&swLang=8&mode=2&taskId=135&swItem=MTX-a94c23fca5ac4a21a3b88aa931

If you are using Gen 8 blades you need at least firmware version 3.51. VC 3.51 can be downloaded from: http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=3709945&prodSeriesId=3794423&prodNameId=3794431&swEnvOID=4091&swLang=8&taskId=135&swItem=MTX-bd78dea478694b249c2a637281&mode=3

If you are still using firmware pre-3.17 you need to be aware of two major issues:

HP Flex-10 issue and DNS causing network outages.

If you are using pre-3.17 firmware, see my post to implement a DNS workaround for a serious issue that can cause the Flex-10 switches to lose connectivity with each other.

HP Flex-10 3.15 firmware bug and PXE kills Virtual Connect.

If you are using 3.15 firmware, see my post about ensuring all your unused blades are powered off to avoid a bug which causes the Flex-10 switches to restart after a memory leak caused by blades continually PXE booting.

Blade Server Firmware

HP Service Pack for ProLiant

HP changed the way it does firmware and drivers and opted for an integrated approach which it is calling HP Service Pack for ProLiant (SPP) . The idea is that 3-5 times a year HP will release a whole pack of firmware and software which should all be tested together for compatibility.

You will be able to use HP Smart Update Manager (HP SUM) to look at an entire environment from HP Blades through to the Chassis OA & Virtual Connect and be able to see all interdependencies and update firmware across the whole lot in the correct order. These will now be named by the date they were released such as 2012.02.0 and hopefully always be downloadable from:

http://h18004.www1.hp.com/products/servers/service_packs/en/index.html

These SPPs will be supported for 1 year which means theoretically HP won’t force you to always upgrade to the latest drivers and firmware unless they can show that the problem you are experiencing is due to a firmware / software problem. There’s a whole bunch of documentation here which you can use to read up on it.

Unfortunately the SPP software isn’t supported for all operating systems but will be the way going forward. OS Support is for: Windows Server 2008 and later, RHEL 5 and later and SLES 10 and later.

If you are using ESXi however, you will not be actually installing any SPP software on the host but just using the firmware components so don’t get too bogged down by OS supportability.

The latest SPP is 2013.09 and can be downloaded from: http://h18004.www1.hp.com/products/servers/service_packs/en/index.html

From the 2013.02 release you can also have online updating for particular server models of System ROM, iLO, SmartArray Controller and Drive firmware with ESXi 5.x using what HP are calling Smart Components which can also use Update Manager giving you a much simpler firmware update process for the supported firmware packages.

You can see the full supported list here: http://vibsdepot.hp.com/hpq/SmartComponents/

Smart Update Firmware DVD

If you are not using ESXi or OS versions on the SPP support list you need to use the old Smart Update Firmware DVD.

The current updated version is 9.30 (2011.05.05)

Update your blade server firmware to the automatic deployment DVD version 9 or your Nics may not be visible.

Version 8.x of the firmware DVD required you to manually update components which is a pain. Boot off the DVD and it does everything automatically. Unfortunately if you run it through an iLO connection it disconnects during the update so just run it again to ensure it runs through. The host will just loop through reboot after automatic installation after reboot so once it’s been through the process the second time you are done and can disconnect the iLO DVD Drive.

You can download 9.30 (2011.05.05)  from:

http://h20566.www2.hp.com/portal/site/hpsc/template.PAGE/public/psi/swdDetails/?sp4ts.oid=3884099&spf_p.tpst=swdMain&spf_p.prp_swdMain=wsrp-navigationalState%3Didx%253D%257CswItem%253DMTX_94560dcc34e845edbb13a26792%257CswEnvOID%253D1005%257CitemLocale%253D%257CswLang%253D%257Cmode%253D%257Caction%253DdriverDocument&javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken

ESX(i) Nic Driver & Firmware Versions

Gen 8 and G7 blades use an Emulex Nic. ESXi versions up to 5.1 have the driver called  be2net. ESXi 5.5 has a  new drive model and so the Emulex drivers are now called elxnet.

G6 blades use a Broadcom bnx2x Nic for all ESXi versions.

There are two separate but related components that you need to ensure are in sync. The Nic driver software version that runs in ESXi and the Nic firmware software version which is updated separately.

HP creates ESX(i) bundles which contain supported HP software versions for various ESX(i) versions. As of Feb 2013 you can also update some firmware online from ESXi but this doesn’t include Nic firmware.

Nic Driver software can be downloaded as a .VIB file and be uploaded to Update Manager to patch your hosts or if you are using Auto Deploy you can build the driver into a custom ESXi 5.x depot using VMware Image Builder.

To manually update each host you can enable ESXi troubleshooting mode as well as SSH. Extract the .vib file and copy it to the local /tmp folder and use esxcli to install.

Example:

esxcli software vib install -v /tmp/net-be2net-4.0.355.1-1OEM.500.0.0.406165.x86_64.vib

VMware ESXi 5.5

Gen8 / G7 Blades

You need to be using Emulex elxnet nic driver version 10.0.575.7  which is included in the HP Custom Image for VMware ESXi 5.5 September 2013 or can be downloaded from VMware.

You need to be using Emulex firmware version 4.6.247.5 which can be updated using the Emulex OneConnect-Flash .ISO.

There is a problem with Emulex firmware prior to version 4.1.450.7 that can result in SmartLink otherwise known as Device Control Channel (DCC) not working with a 10Gb physical link when you have full height blades. When you upgrade the OA, you may lose network connectivity. Update the NIC firmware and look at Customer Advisory c03600027 before upgrading the OA.

G6 Blades

You need to be using Broadcom bnx2x nic driver version 1.78.28.v55.2 which is included in the HP Custom Image for VMware ESXi 5.5 September 2013 or can be downloaded from VMware.

You need to be using Broadcom firmware version 6.2.26  which can be updated using the HP Service Pack for ProLiant version 2013.09.0(b)

VMware ESXi 5.1

Gen8 / G7 Blades

You need to be using Emulex be2net nic driver version 4.6.247.10 which is included in the HP Custom Image for VMware ESXi 5.1.0 Update 1 September 2013 or can be downloaded separately from VMware.

You need to be using Emulex firmware version 4.6.247.5 which can be updated using the Emulex OneConnect-Flash .ISO.

There is a problem with Emulex firmware prior to version 4.1.450.7 that can result in SmartLink otherwise known as Device Control Channel (DCC) not working with a 10Gb physical link when you have full height blades. When you upgrade the OA, you may lose network connectivity. Update the NIC firmware and look at Customer Advisory c03600027 before upgrading the OA.

G6 Blades

You need to be using Broadcom bnx2x nic driver version 1.74.17.v50.1 which is included in the HP Custom Image for VMware ESXi 5.1.0 Update 1 September 2013 or can be downloaded separately from VMware

You need to be using Broadcom firmware version 6.2.26  which can be updated using the HP Service Pack for ProLiant version 2013.09.0(b)

VMware ESXi 5.0

Gen8 / G7 Blades

The inbox ESXi5.0 Emulex be2net driver 4.0.88.0 doesn’t work if you used VLAN tagging anywhere in your network path and also has issues with PXE which will affect you if you wanted to use Auto Deploy.

You need to be using Emulex be2net nic driver version 4.6.247.07 which is included in the HP Custom Image for VMware ESXi 5.0.0 Update 3 October 2013 or can be downloaded separately from VMware.

You need to be using Emulex firmware version 4.6.247.5 which can be updated using the Emulex OneConnect-Flash .ISO.

There is a problem with Emulex firmware prior to version 4.1.450.7 that can result in SmartLink otherwise known as Device Control Channel (DCC) not working with a 10Gb physical link when you have full height blades. When you upgrade the OA, you may lose network connectivity. Update the NIC firmware and look at Customer Advisory c03600027 before upgrading the OA.

G6 Blades

You need to be using Broadcom bnx2x nic driver version 4.2.327.0 which is included in the HP Custom Image for VMware ESXi 5.0.0 Update 2 February 2013: VMware-ESXi-5.0.0-Update2-914586-HP-5.27.20.iso or can be downloaded separately from VMware

You need to be using Broadcom firmware version 6.2.26  which can be updated using the HP Service Pack for ProLiant version 2013.09.0(b)

If you are having any issues with previous drivers:

After the inbox driver, HP released a new driver version 4.0.355.1 which they then pulled as there were still issues with this driver and PXE.

HP then re-released the same driver but with a new Nic firmware requirement version 4.0.360.7. Although the numbers look very similar, 4.0.355.1 is the Nic driver version and 4.0.360.7 is the Nic firmware version.

HP then released a newer 4.0.360.20 firmware which was then pulled and replaced with 4.0.360.15a. Yes, I know .15a surely shouldn’t be released after .20!

There was however a potential known issue with the 4.0.360.15a firmware update procedure which is listed in the installation instructions:

If you receive the following error during the upgrade “Warning: One or more CRC errors was detected during the flash operation. Prior to restarting your system it is important to retry the flash update. A failed flash update may leave your system in an unknown state”

Perform the following for the adapters listed:

• BL460 or NC553i install the 2.102.517.703 iso following the above steps, reboot and reinstall the 4.0.xxx.xx.iso image
• CN1000E or NC550SFP install the 2.102.517.708 iso following the above steps, reboot and reinstall the 4.0.xxx.xx.iso image

VMware ESX/ESX(i) 4.1 Update 1/2

Gen8 / G7 Blades

You need the Emulex be2net driver version 4.1.334.0 which can be downloaded from here: http://downloads.vmware.com/d/details/dt_esxi40_emulex_be2net_403060/ZHcqYnQqaGpiZEBlZA

You need the Emulex firmware version 4.1.402.20 which can be downloaded from here: http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=lamerica_nsc_carib&prodTypeId=329290&prodSeriesId=4324629&swItem=co-106538-1&prodNameId=4324630&swEnvOID=54&swLang=8&taskId=135&mode=4&idx=1

There is a problem with Emulex firmware prior to version 4.1.450.7 that can result in SmartLink otherwise known as Device Control Channel (DCC) not working with a 10Gb physical link when you have full height blades. When you upgrade the OA, you may lose network connectivity. Update the NIC firmware and look at Customer Advisory c03600027 before upgrading the OA.

G6 Blades

HP and VMware have finally tested and certified a driver that works with DCC/SmartLink and is available through Update Manager as the shipped ESX 4.1 drivers caused PSODs.

The latest driver which supports DCC/SmartLink for ESX 4.1 is 1.70.34.v41.1

This driver can be downloaded from VMware and deployed using Update Manager.

If you are upgrading from the previously supported driver 1.60.50.v41.2 the heap memory fix is included so you don’t need to remove the workaround as was described in the original issue. http://kb.vmware.com/kb/1031805

VMware ESX/ESX i 4.0

Gen8 / G7 Blades You need the Emulex be2net driver version 4.1.334.0 which can be downloaded from here: http://downloads.vmware.com/d/details/dt_esxi40_emulex_be2net_403060/ZHcqYnQqaGpiZEBlZA

You need the Emulex firmware version 4.1.402.20 which can be downloaded from here: http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=lamerica_nsc_carib&prodTypeId=329290&prodSeriesId=4324629&swItem=co-106538-1&prodNameId=4324630&swEnvOID=54&swLang=8&taskId=135&mode=4&idx=1

G6 Blades

There is a long standing issue with the ESX Broadcom bnx2 driver. There have been quite a few drivers released, some of which support DCC/SmartLink and some that don’t.

The latest driver which supports DCC/SmartLink for ESX 4.0 is 1.62.11.v40.1

This driver can be downloaded from VMware and deployed using Update Manager.