Updated: 28 October 2013 to include new HP September bundles and release of ESXi 5.5.
Many companies are currently going through major infrastructure changes as the reach of virtualisation into IT increases and they start moving towards a cloud model.
Blade server use is growing as people start to think more seriously about large server farms and building the infrastructure for a “cloud”, be it internal(private) or external (public). HP has been doing a lot of work selling its blades (BL460C and BL490C) in C7000 chassis with Flex-10 switches.
These infrastructure changes can be pretty complicated to implement as you need to plan for a whole new server hosting environment, not just adding another server into a rack somewhere.
In a converged world you also need to get server people, power people, VM people, storage people and networking people to work together which is a challenge at the best of times in traditional siloed enterprise IT departments.
HP has however had some problems over the years delivering the technical message to get this all done. Unfortunately there often wasn’t enough clear information from HP about what needs to be done. Their reference architectures and Flex-10 cookbook were incomplete or out of date and in some cases contradictory and misleading. This has improved somewhat over the years but it doesn’t help that HP’s website must rate as one of the worst to find anything and it’s an IT company…quite a big one as well and it lets itself down when its hardware in my opinion is fantastic.
So, here’s a pre-requisite list I’ve been building up and maintaining over the years to hopefully get you on your way.
Onboard Administrator Firmware
HP BladeSystem c-Class Onboard Administrator (OA) Firmware
Update all your chassis OA firmware to 4.01. Although this is a v4 release the update is not nearly as dramatic as Virtual Connect 4. The Onboard Administrator upgrade from 3.71 to 4.01 just adds IPv6 support, new hardware models and a raft of bug fixes.
Update all your chassis OA firmware to 3.71.
http://h20566.www2.hp.com/portal/site/hpsc/template.PAGE/public/psi/swdDetails/?sp4ts.oid=3692952&spf_p.tpst=swdMain&spf_p.prp_swdMain=wsrp-navigationalState%3Didx%253D0%257CswItem%253DMTX_b9dd1a1e38b8457498123b9c52%257CswEnvOID%253D4064%257CitemLocale%253D%257CswLang%253D%257Cmode%253D3%257Caction%253DdriverDocument&javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken
Older version information:
There is a problem with Emulex firmware prior to version 4.1.450.7 that can result in SmartLink otherwise known as Device Control Channel (DCC) not working with a 10Gb physical link when you have full height blades. When you upgrade the OA, you may lose network connectivity. Update the NIC firmware and look at Customer Advisory c03600027 before upgrading the OA.
If you are using Gen 8 blades you will have to update your OA firmware to at least 3.50 which is the latest Recommended release as although the blades will be recognised there will be issues with Virtual Connect.
A newer version 3.55 was released the next day which fixed some authentication and EBIPA issues which must have slipped through the testing net with 3.50 (great!) . OA 3.50 can be downloaded from: http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=3709945&prodSeriesId=1844065&prodNameId=3692952&swEnvOID=4064&swLang=8&taskId=135&swItem=MTX-00b278b5e9194e7ab0ab362d55&mode=3
OA 3.55 can be downloaded from:
http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=3709945&prodSeriesId=1844065&prodNameId=3692952&swEnvOID=4064&swLang=8&taskId=135&swItem=MTX-b22a6014742b4bd7a0a05557b9&mode=3
Virtual Connect Firmware
HP BladeSystem c-Class Virtual Connect Ethernet (VC-Enet), VC 4Gb FC, VC 8Gb 20-port FC, and VC 8Gb 24-port FC modules
Update your Virtual Connect Flex-10 switches to firmware 3.75 as this is a Recommended release with many bug fixes.
Virtual Connect 4.01 has been released which includes many more features and has also since been updated to 4.1 . It’s still relatively new so if you are going to use it, do your testing.
Upgrade Alert: If you have both single dense and double dense servers in your chassis and you are upgrading from Virtual Connect firmwares 3.10-3.60 up to Virtual Connect firmwares 3.70-3.75 you need to be aware of a serious issue where upgrading the firmware may cause your server profiles to be unassigned which will cause a network outage.
See this post: HP Virtual Connect Firmware upgrade may cause Server Profiles to be unassigned causing a network outage
VC 3.75 can be downloaded from: http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=3709945&prodSeriesId=3794423&prodNameId=3794431&swEnvOID=4091&swLang=8&mode=2&taskId=135&swItem=MTX-a94c23fca5ac4a21a3b88aa931
If you are using Gen 8 blades you need at least firmware version 3.51. VC 3.51 can be downloaded from: http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=3709945&prodSeriesId=3794423&prodNameId=3794431&swEnvOID=4091&swLang=8&taskId=135&swItem=MTX-bd78dea478694b249c2a637281&mode=3
If you are still using firmware pre-3.17 you need to be aware of two major issues:
HP Flex-10 issue and DNS causing network outages.
If you are using pre-3.17 firmware, see my post to implement a DNS workaround for a serious issue that can cause the Flex-10 switches to lose connectivity with each other.
HP Flex-10 3.15 firmware bug and PXE kills Virtual Connect.
If you are using 3.15 firmware, see my post about ensuring all your unused blades are powered off to avoid a bug which causes the Flex-10 switches to restart after a memory leak caused by blades continually PXE booting.
Blade Server Firmware
HP Service Pack for ProLiant
HP changed the way it does firmware and drivers and opted for an integrated approach which it is calling HP Service Pack for ProLiant (SPP) . The idea is that 3-5 times a year HP will release a whole pack of firmware and software which should all be tested together for compatibility.
You will be able to use HP Smart Update Manager (HP SUM) to look at an entire environment from HP Blades through to the Chassis OA & Virtual Connect and be able to see all interdependencies and update firmware across the whole lot in the correct order. These will now be named by the date they were released such as 2012.02.0 and hopefully always be downloadable from:
http://h18004.www1.hp.com/products/servers/service_packs/en/index.html
These SPPs will be supported for 1 year which means theoretically HP won’t force you to always upgrade to the latest drivers and firmware unless they can show that the problem you are experiencing is due to a firmware / software problem. There’s a whole bunch of documentation here which you can use to read up on it.
Unfortunately the SPP software isn’t supported for all operating systems but will be the way going forward. OS Support is for: Windows Server 2008 and later, RHEL 5 and later and SLES 10 and later.
If you are using ESXi however, you will not be actually installing any SPP software on the host but just using the firmware components so don’t get too bogged down by OS supportability.
The latest SPP is 2013.09 and can be downloaded from: http://h18004.www1.hp.com/products/servers/service_packs/en/index.html
From the 2013.02 release you can also have online updating for particular server models of System ROM, iLO, SmartArray Controller and Drive firmware with ESXi 5.x using what HP are calling Smart Components which can also use Update Manager giving you a much simpler firmware update process for the supported firmware packages.
You can see the full supported list here: http://vibsdepot.hp.com/hpq/SmartComponents/
Smart Update Firmware DVD
If you are not using ESXi or OS versions on the SPP support list you need to use the old Smart Update Firmware DVD.
The current updated version is 9.30 (2011.05.05)
Update your blade server firmware to the automatic deployment DVD version 9 or your Nics may not be visible.
Version 8.x of the firmware DVD required you to manually update components which is a pain. Boot off the DVD and it does everything automatically. Unfortunately if you run it through an iLO connection it disconnects during the update so just run it again to ensure it runs through. The host will just loop through reboot after automatic installation after reboot so once it’s been through the process the second time you are done and can disconnect the iLO DVD Drive.
You can download 9.30 (2011.05.05) from:
http://h20566.www2.hp.com/portal/site/hpsc/template.PAGE/public/psi/swdDetails/?sp4ts.oid=3884099&spf_p.tpst=swdMain&spf_p.prp_swdMain=wsrp-navigationalState%3Didx%253D%257CswItem%253DMTX_94560dcc34e845edbb13a26792%257CswEnvOID%253D1005%257CitemLocale%253D%257CswLang%253D%257Cmode%253D%257Caction%253DdriverDocument&javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken
ESX(i) Nic Driver & Firmware Versions
Gen 8 and G7 blades use an Emulex Nic. ESXi versions up to 5.1 have the driver called be2net. ESXi 5.5 has a new drive model and so the Emulex drivers are now called elxnet.
G6 blades use a Broadcom bnx2x Nic for all ESXi versions.
There are two separate but related components that you need to ensure are in sync. The Nic driver software version that runs in ESXi and the Nic firmware software version which is updated separately.
HP creates ESX(i) bundles which contain supported HP software versions for various ESX(i) versions. As of Feb 2013 you can also update some firmware online from ESXi but this doesn’t include Nic firmware.
Nic Driver software can be downloaded as a .VIB file and be uploaded to Update Manager to patch your hosts or if you are using Auto Deploy you can build the driver into a custom ESXi 5.x depot using VMware Image Builder.
To manually update each host you can enable ESXi troubleshooting mode as well as SSH. Extract the .vib file and copy it to the local /tmp folder and use esxcli to install.
Example:
esxcli software vib install -v /tmp/net-be2net-4.0.355.1-1OEM.500.0.0.406165.x86_64.vib
VMware ESXi 5.5
Gen8 / G7 Blades
You need to be using Emulex elxnet nic driver version 10.0.575.7 which is included in the HP Custom Image for VMware ESXi 5.5 September 2013 or can be downloaded from VMware.
You need to be using Emulex firmware version 4.6.247.5 which can be updated using the Emulex OneConnect-Flash .ISO.
There is a problem with Emulex firmware prior to version 4.1.450.7 that can result in SmartLink otherwise known as Device Control Channel (DCC) not working with a 10Gb physical link when you have full height blades. When you upgrade the OA, you may lose network connectivity. Update the NIC firmware and look at Customer Advisory c03600027 before upgrading the OA.
G6 Blades
You need to be using Broadcom bnx2x nic driver version 1.78.28.v55.2 which is included in the HP Custom Image for VMware ESXi 5.5 September 2013 or can be downloaded from VMware.
You need to be using Broadcom firmware version 6.2.26 which can be updated using the HP Service Pack for ProLiant version 2013.09.0(b)
VMware ESXi 5.1
Gen8 / G7 Blades
You need to be using Emulex be2net nic driver version 4.6.247.10 which is included in the HP Custom Image for VMware ESXi 5.1.0 Update 1 September 2013 or can be downloaded separately from VMware.
You need to be using Emulex firmware version 4.6.247.5 which can be updated using the Emulex OneConnect-Flash .ISO.
There is a problem with Emulex firmware prior to version 4.1.450.7 that can result in SmartLink otherwise known as Device Control Channel (DCC) not working with a 10Gb physical link when you have full height blades. When you upgrade the OA, you may lose network connectivity. Update the NIC firmware and look at Customer Advisory c03600027 before upgrading the OA.
G6 Blades
You need to be using Broadcom bnx2x nic driver version 1.74.17.v50.1 which is included in the HP Custom Image for VMware ESXi 5.1.0 Update 1 September 2013 or can be downloaded separately from VMware
You need to be using Broadcom firmware version 6.2.26 which can be updated using the HP Service Pack for ProLiant version 2013.09.0(b)
VMware ESXi 5.0
Gen8 / G7 Blades
The inbox ESXi5.0 Emulex be2net driver 4.0.88.0 doesn’t work if you used VLAN tagging anywhere in your network path and also has issues with PXE which will affect you if you wanted to use Auto Deploy.
You need to be using Emulex be2net nic driver version 4.6.247.07 which is included in the HP Custom Image for VMware ESXi 5.0.0 Update 3 October 2013 or can be downloaded separately from VMware.
You need to be using Emulex firmware version 4.6.247.5 which can be updated using the Emulex OneConnect-Flash .ISO.
There is a problem with Emulex firmware prior to version 4.1.450.7 that can result in SmartLink otherwise known as Device Control Channel (DCC) not working with a 10Gb physical link when you have full height blades. When you upgrade the OA, you may lose network connectivity. Update the NIC firmware and look at Customer Advisory c03600027 before upgrading the OA.
G6 Blades
You need to be using Broadcom bnx2x nic driver version 4.2.327.0 which is included in the HP Custom Image for VMware ESXi 5.0.0 Update 2 February 2013: VMware-ESXi-5.0.0-Update2-914586-HP-5.27.20.iso or can be downloaded separately from VMware
You need to be using Broadcom firmware version 6.2.26 which can be updated using the HP Service Pack for ProLiant version 2013.09.0(b)
If you are having any issues with previous drivers:
After the inbox driver, HP released a new driver version 4.0.355.1 which they then pulled as there were still issues with this driver and PXE.
HP then re-released the same driver but with a new Nic firmware requirement version 4.0.360.7. Although the numbers look very similar, 4.0.355.1 is the Nic driver version and 4.0.360.7 is the Nic firmware version.
HP then released a newer 4.0.360.20 firmware which was then pulled and replaced with 4.0.360.15a. Yes, I know .15a surely shouldn’t be released after .20!
There was however a potential known issue with the 4.0.360.15a firmware update procedure which is listed in the installation instructions:
If you receive the following error during the upgrade “Warning: One or more CRC errors was detected during the flash operation. Prior to restarting your system it is important to retry the flash update. A failed flash update may leave your system in an unknown state”
Perform the following for the adapters listed:
- BL460 or NC553i install the 2.102.517.703 iso following the above steps, reboot and reinstall the 4.0.xxx.xx.iso image
- CN1000E or NC550SFP install the 2.102.517.708 iso following the above steps, reboot and reinstall the 4.0.xxx.xx.iso image
VMware ESX/ESX(i) 4.1 Update 1/2
Gen8 / G7 Blades
You need the Emulex be2net driver version 4.1.334.0 which can be downloaded from here: http://downloads.vmware.com/d/details/dt_esxi40_emulex_be2net_403060/ZHcqYnQqaGpiZEBlZA
You need the Emulex firmware version 4.1.402.20 which can be downloaded from here: http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=lamerica_nsc_carib&prodTypeId=329290&prodSeriesId=4324629&swItem=co-106538-1&prodNameId=4324630&swEnvOID=54&swLang=8&taskId=135&mode=4&idx=1
There is a problem with Emulex firmware prior to version 4.1.450.7 that can result in SmartLink otherwise known as Device Control Channel (DCC) not working with a 10Gb physical link when you have full height blades. When you upgrade the OA, you may lose network connectivity. Update the NIC firmware and look at Customer Advisory c03600027 before upgrading the OA.
G6 Blades
HP and VMware have finally tested and certified a driver that works with DCC/SmartLink and is available through Update Manager as the shipped ESX 4.1 drivers caused PSODs.
The latest driver which supports DCC/SmartLink for ESX 4.1 is 1.70.34.v41.1
This driver can be downloaded from VMware and deployed using Update Manager.
If you are upgrading from the previously supported driver 1.60.50.v41.2 the heap memory fix is included so you don’t need to remove the workaround as was described in the original issue. http://kb.vmware.com/kb/1031805
VMware ESX/ESX i 4.0
Gen8 / G7 Blades You need the Emulex be2net driver version 4.1.334.0 which can be downloaded from here: http://downloads.vmware.com/d/details/dt_esxi40_emulex_be2net_403060/ZHcqYnQqaGpiZEBlZA
You need the Emulex firmware version 4.1.402.20 which can be downloaded from here: http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=lamerica_nsc_carib&prodTypeId=329290&prodSeriesId=4324629&swItem=co-106538-1&prodNameId=4324630&swEnvOID=54&swLang=8&taskId=135&mode=4&idx=1
G6 Blades
There is a long standing issue with the ESX Broadcom bnx2 driver. There have been quite a few drivers released, some of which support DCC/SmartLink and some that don’t.
The latest driver which supports DCC/SmartLink for ESX 4.0 is 1.62.11.v40.1
This driver can be downloaded from VMware and deployed using Update Manager.
Recent Comments