WoodITWork.com

I’ve had confirmation from HP that the ESX 4.1 updated Broadcom 1.60 driver DOES support DCC/SmartLink.

I had queried this in my previous post HP Flex-10, ESX and Broadcom fun continues…

So, looks like the 1.60 driver is the one to go for on ESX 4.1 as long as you have the Flex-10 firmware prerequisites and update the Heap Memory allocated by the Broadcom bnx2x driver as listed here:
http://kb.vmware.com/kb/1031805

Unfortunately this driver isn’t available through Update Manager but follow my previous post and you can use PowerCLI.

I’ve also updated my Flex-10 ESX pre-requisites post so you have all the information in one place.

HP also told me they plan to update their original advisory and get better communication on the VMware site to mention DCC/SmartLink, let’s hope this happens.

I’ve been updating my HP Flex-10 firmware information to reflect what’s been happening with ESX 4.1 and it looks like the fun is still ongoing.

My Flex-10 ESX pre-requisites post has also been updated with the latest information.

ESX 4.1 was released with an inbox Broadcom bnx2x driver 1.54.1.v41.1-1vmw

This driver was found to cause Network connectivity issues and PSODs which I warned about and at that stage recommended not deploying ESX 4.1 if you had affected Nics.

VMWare’s KB article: http://kb.vmware.com/kb/1029368 – VMware ESX/ESXi 4.1 host with Broadcom bnx2x (in-box driver version 1.54.1.v41.1-1vmw) experiences a loss of network connectivity and a purple diagnostic screen.

This driver also unfortunately took a few steps back by removing DCC/SmartLink which I mentioned in this post.

Read more…

VMware has released a new Broadcom bnx2x driver today and made it available through Update Manager.
http://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&externalId=1029394

The update looks like it fixes the critical issues referenced in my previous post.

What is very strange though is the 4.1 bundled version number was:
1.54.1.v41.1-1vmw
The new version number is:
1.54.1.v41.1-2vmw

See the difference, a small little 2. So this looks like a very minor release. What is the 1.60 driver all about then?

Does this minor 1.54-2 release fix the issues with the built in driver but not fix SmartLink / DDC. Does the 1.60 driver fix the issues with the built-in driver or just fix SmartLink / DDC?

I haven’t been able to find any more information on what the 1.60 driver fixes or breaks as there doesn’t seem to be any published information about it.

I’m very happy these drivers are now being released through Update Manager which is the easiest deployment method although you can still use PowerCLI to deploy them but why release 1.60 and then release 1.54-2 later?

I’ve said it before: Come on everyone…VMware, Broadcom, HP need to get together and sort this out!

I generally avoid having to use the vSphere CLI by doing whatever I can in PowerCLI.

If you are primarily a Unix shop then the CLI is probably your scripting environment of choice.

If you run ESXi you also don’t have the use of the Service Console to run commands so can fire up the vSphere Management Assistant (vMA) and using the CLI can run commands remotely against the ESXi host. This is especially useful for looking at performance stats with resxtop.

Here’s a good command line reference on VMware’s site updated to 4.1.
http://www.vmware.com/support/developer/vcli/vcli41/doc/reference/index.html

Time to finish up all those conversions before it could be too late!

We all know that Microsoft Windows NT and Windows 2000 are no longer supported by Microsoft but there are still many many instances out there begging for an upgrade but running applications which just cannot easily be moved or are working fine so people don’t see a reason to upgrade.

Often these servers are also running on some very old hardware so converting them to VMs at least allows you to move off the old hardware but making them VMs using VMware’s P2V tool is going to get harder as time goes by.

VMware vCenter Converter Standalone 4.3 has dropped support for Windows NT and 2000.

It’s all there in the release notes.

You can still use the old Converter 4.0.1 to P2V your Windows NT and 2000 servers but this may be not supported by VMware depending on what ESX version you are converting to.

You can use 4.0.1 to convert to ESX 4.0 but not to ESX 4.1 although I can’t see why this wouldn’t actually work despite it not being supported.

If you need a supported conversion, hopefully you haven’t been very keen and upgraded all your hosts to 4.1 yet. You could in theory do a two stage process, migrate using Converter 4.0.1 to ESX 4.0 and then migrate the VM to 4.1.

If all your hosts are 4.1 well you may just have to be brave and go for it!

VMware has just released an advisory for the Broadcom bnx2x Inbox drivers in ESX 4.1 which will affect HP Servers including blades.

ESX/ESXi 4.1 with Broadcom bnx2x Inbox driver version 1.54.1.v41.1-1vmw experiences a loss of network connectivity and a purple diagnostic screen

http://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&externalId=1029368

The resolution (which is only a workaround) says it manifests itself when using the IP Checksum feature which is on by default on ESX and ESXi 4.1. This feature moves the checksumming (if there’s such a word) from the OS stack to the adapter and can cause a driver or firmware panic.

Now, the question is, is this a serious enough problem to immediately go out and disable checksumming?  The KB doesn’t say but this looks like it could be a serious issue.

Hopefully some more information will be forthcoming from VMware / HP / Broadcom but at least you’ve been warned!

Updated: 28 October 2013 to include new HP September bundles and release of ESXi 5.5.

Many companies are currently going through major infrastructure changes as the reach of virtualisation into IT increases and they start moving towards a cloud model.

Blade server use is growing as people start to think more seriously about large server farms and building the infrastructure for a “cloud”, be it internal(private) or external (public). HP has been doing a lot of work selling its blades (BL460C and BL490C) in C7000 chassis with Flex-10 switches.

These infrastructure changes can be pretty complicated to implement as you need to plan for a whole new server hosting environment, not just adding another server into a rack somewhere.

In a converged world you also need to get server people, power people, VM people, storage people and networking people to work together which is a challenge at the best of times in traditional siloed enterprise IT departments.

HP has however had some problems over the years delivering the technical message to get this all done.  Unfortunately there often wasn’t enough clear information from HP about what needs to be done. Their reference architectures and Flex-10 cookbook were incomplete or out of date and in some cases contradictory and misleading. This has improved somewhat over the years but it doesn’t help that HP’s website must rate as one of the worst to find anything and it’s an IT company…quite a big one as well and it lets itself down when its hardware in my opinion is fantastic.

So, here’s a pre-requisite list I’ve been building up and maintaining over the years to hopefully get you on your way.

Onboard Administrator Firmware

HP BladeSystem c-Class Onboard Administrator (OA) Firmware

Update all your chassis OA firmware to 4.01. Although this is a v4 release the update is not nearly as dramatic as Virtual Connect 4. The Onboard Administrator upgrade from 3.71 to 4.01 just adds IPv6 support, new hardware models and a raft of bug fixes.

Update all your chassis OA firmware to 3.71.

http://h20566.www2.hp.com/portal/site/hpsc/template.PAGE/public/psi/swdDetails/?sp4ts.oid=3692952&spf_p.tpst=swdMain&spf_p.prp_swdMain=wsrp-navigationalState%3Didx%253D0%257CswItem%253DMTX_b9dd1a1e38b8457498123b9c52%257CswEnvOID%253D4064%257CitemLocale%253D%257CswLang%253D%257Cmode%253D3%257Caction%253DdriverDocument&javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken

Older version information:

There is a problem with Emulex firmware prior to version 4.1.450.7 that can result in SmartLink otherwise known as Device Control Channel (DCC) not working with a 10Gb physical link when you have full height blades. When you upgrade the OA, you may lose network connectivity. Update the NIC firmware and look at Customer Advisory c03600027 before upgrading the OA.

If you are using Gen 8 blades you will have to update your OA firmware to at least 3.50 which is the latest Recommended release as although the blades will be recognised there will be issues with Virtual Connect.

A newer version 3.55 was released the next day which fixed some authentication and EBIPA issues which must have slipped through the testing net with 3.50 (great!) . OA 3.50 can be downloaded from: http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=3709945&prodSeriesId=1844065&prodNameId=3692952&swEnvOID=4064&swLang=8&taskId=135&swItem=MTX-00b278b5e9194e7ab0ab362d55&mode=3

OA 3.55 can be downloaded from:

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=3709945&prodSeriesId=1844065&prodNameId=3692952&swEnvOID=4064&swLang=8&taskId=135&swItem=MTX-b22a6014742b4bd7a0a05557b9&mode=3

Virtual Connect Firmware

HP BladeSystem c-Class Virtual Connect Ethernet (VC-Enet), VC 4Gb FC, VC 8Gb 20-port FC, and VC 8Gb 24-port FC modules

Update your Virtual Connect Flex-10 switches to firmware 3.75 as this is a Recommended release with many bug fixes.

Virtual Connect 4.01 has been released which includes many more features and has also since been updated to 4.1 . It’s still relatively new so if you are going to use it, do your testing.

Upgrade Alert: If you have both single dense and double dense servers in your chassis and you are upgrading from Virtual Connect firmwares 3.10-3.60 up to Virtual Connect firmwares 3.70-3.75 you need to be aware of a serious issue where upgrading the firmware may cause your server profiles to be unassigned which will cause a network outage.

See this post: HP Virtual Connect Firmware upgrade may cause Server Profiles to be unassigned causing a network outage

VC 3.75 can be downloaded from: http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=3709945&prodSeriesId=3794423&prodNameId=3794431&swEnvOID=4091&swLang=8&mode=2&taskId=135&swItem=MTX-a94c23fca5ac4a21a3b88aa931

If you are using Gen 8 blades you need at least firmware version 3.51. VC 3.51 can be downloaded from: http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=3709945&prodSeriesId=3794423&prodNameId=3794431&swEnvOID=4091&swLang=8&taskId=135&swItem=MTX-bd78dea478694b249c2a637281&mode=3

If you are still using firmware pre-3.17 you need to be aware of two major issues:

HP Flex-10 issue and DNS causing network outages.

If you are using pre-3.17 firmware, see my post to implement a DNS workaround for a serious issue that can cause the Flex-10 switches to lose connectivity with each other.

HP Flex-10 3.15 firmware bug and PXE kills Virtual Connect.

If you are using 3.15 firmware, see my post about ensuring all your unused blades are powered off to avoid a bug which causes the Flex-10 switches to restart after a memory leak caused by blades continually PXE booting.

Blade Server Firmware

HP Service Pack for ProLiant

HP changed the way it does firmware and drivers and opted for an integrated approach which it is calling HP Service Pack for ProLiant (SPP) . The idea is that 3-5 times a year HP will release a whole pack of firmware and software which should all be tested together for compatibility.

You will be able to use HP Smart Update Manager (HP SUM) to look at an entire environment from HP Blades through to the Chassis OA & Virtual Connect and be able to see all interdependencies and update firmware across the whole lot in the correct order. These will now be named by the date they were released such as 2012.02.0 and hopefully always be downloadable from:

http://h18004.www1.hp.com/products/servers/service_packs/en/index.html

These SPPs will be supported for 1 year which means theoretically HP won’t force you to always upgrade to the latest drivers and firmware unless they can show that the problem you are experiencing is due to a firmware / software problem. There’s a whole bunch of documentation here which you can use to read up on it.

Unfortunately the SPP software isn’t supported for all operating systems but will be the way going forward. OS Support is for: Windows Server 2008 and later, RHEL 5 and later and SLES 10 and later.

If you are using ESXi however, you will not be actually installing any SPP software on the host but just using the firmware components so don’t get too bogged down by OS supportability.

The latest SPP is 2013.09 and can be downloaded from: http://h18004.www1.hp.com/products/servers/service_packs/en/index.html

From the 2013.02 release you can also have online updating for particular server models of System ROM, iLO, SmartArray Controller and Drive firmware with ESXi 5.x using what HP are calling Smart Components which can also use Update Manager giving you a much simpler firmware update process for the supported firmware packages.

You can see the full supported list here: http://vibsdepot.hp.com/hpq/SmartComponents/

Smart Update Firmware DVD

If you are not using ESXi or OS versions on the SPP support list you need to use the old Smart Update Firmware DVD.

The current updated version is 9.30 (2011.05.05)

Update your blade server firmware to the automatic deployment DVD version 9 or your Nics may not be visible.

Version 8.x of the firmware DVD required you to manually update components which is a pain. Boot off the DVD and it does everything automatically. Unfortunately if you run it through an iLO connection it disconnects during the update so just run it again to ensure it runs through. The host will just loop through reboot after automatic installation after reboot so once it’s been through the process the second time you are done and can disconnect the iLO DVD Drive.

You can download 9.30 (2011.05.05)  from:

http://h20566.www2.hp.com/portal/site/hpsc/template.PAGE/public/psi/swdDetails/?sp4ts.oid=3884099&spf_p.tpst=swdMain&spf_p.prp_swdMain=wsrp-navigationalState%3Didx%253D%257CswItem%253DMTX_94560dcc34e845edbb13a26792%257CswEnvOID%253D1005%257CitemLocale%253D%257CswLang%253D%257Cmode%253D%257Caction%253DdriverDocument&javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken

ESX(i) Nic Driver & Firmware Versions

Gen 8 and G7 blades use an Emulex Nic. ESXi versions up to 5.1 have the driver called  be2net. ESXi 5.5 has a  new drive model and so the Emulex drivers are now called elxnet.

G6 blades use a Broadcom bnx2x Nic for all ESXi versions.

There are two separate but related components that you need to ensure are in sync. The Nic driver software version that runs in ESXi and the Nic firmware software version which is updated separately.

HP creates ESX(i) bundles which contain supported HP software versions for various ESX(i) versions. As of Feb 2013 you can also update some firmware online from ESXi but this doesn’t include Nic firmware.

Nic Driver software can be downloaded as a .VIB file and be uploaded to Update Manager to patch your hosts or if you are using Auto Deploy you can build the driver into a custom ESXi 5.x depot using VMware Image Builder.

To manually update each host you can enable ESXi troubleshooting mode as well as SSH. Extract the .vib file and copy it to the local /tmp folder and use esxcli to install.

Example:

esxcli software vib install -v /tmp/net-be2net-4.0.355.1-1OEM.500.0.0.406165.x86_64.vib

VMware ESXi 5.5

Gen8 / G7 Blades

You need to be using Emulex elxnet nic driver version 10.0.575.7  which is included in the HP Custom Image for VMware ESXi 5.5 September 2013 or can be downloaded from VMware.

You need to be using Emulex firmware version 4.6.247.5 which can be updated using the Emulex OneConnect-Flash .ISO.

There is a problem with Emulex firmware prior to version 4.1.450.7 that can result in SmartLink otherwise known as Device Control Channel (DCC) not working with a 10Gb physical link when you have full height blades. When you upgrade the OA, you may lose network connectivity. Update the NIC firmware and look at Customer Advisory c03600027 before upgrading the OA.

G6 Blades

You need to be using Broadcom bnx2x nic driver version 1.78.28.v55.2 which is included in the HP Custom Image for VMware ESXi 5.5 September 2013 or can be downloaded from VMware.

You need to be using Broadcom firmware version 6.2.26  which can be updated using the HP Service Pack for ProLiant version 2013.09.0(b)

VMware ESXi 5.1

Gen8 / G7 Blades

You need to be using Emulex be2net nic driver version 4.6.247.10 which is included in the HP Custom Image for VMware ESXi 5.1.0 Update 1 September 2013 or can be downloaded separately from VMware.

You need to be using Emulex firmware version 4.6.247.5 which can be updated using the Emulex OneConnect-Flash .ISO.

There is a problem with Emulex firmware prior to version 4.1.450.7 that can result in SmartLink otherwise known as Device Control Channel (DCC) not working with a 10Gb physical link when you have full height blades. When you upgrade the OA, you may lose network connectivity. Update the NIC firmware and look at Customer Advisory c03600027 before upgrading the OA.

G6 Blades

You need to be using Broadcom bnx2x nic driver version 1.74.17.v50.1 which is included in the HP Custom Image for VMware ESXi 5.1.0 Update 1 September 2013 or can be downloaded separately from VMware

You need to be using Broadcom firmware version 6.2.26  which can be updated using the HP Service Pack for ProLiant version 2013.09.0(b)

VMware ESXi 5.0

Gen8 / G7 Blades

The inbox ESXi5.0 Emulex be2net driver 4.0.88.0 doesn’t work if you used VLAN tagging anywhere in your network path and also has issues with PXE which will affect you if you wanted to use Auto Deploy.

You need to be using Emulex be2net nic driver version 4.6.247.07 which is included in the HP Custom Image for VMware ESXi 5.0.0 Update 3 October 2013 or can be downloaded separately from VMware.

You need to be using Emulex firmware version 4.6.247.5 which can be updated using the Emulex OneConnect-Flash .ISO.

There is a problem with Emulex firmware prior to version 4.1.450.7 that can result in SmartLink otherwise known as Device Control Channel (DCC) not working with a 10Gb physical link when you have full height blades. When you upgrade the OA, you may lose network connectivity. Update the NIC firmware and look at Customer Advisory c03600027 before upgrading the OA.

G6 Blades

You need to be using Broadcom bnx2x nic driver version 4.2.327.0 which is included in the HP Custom Image for VMware ESXi 5.0.0 Update 2 February 2013: VMware-ESXi-5.0.0-Update2-914586-HP-5.27.20.iso or can be downloaded separately from VMware

You need to be using Broadcom firmware version 6.2.26  which can be updated using the HP Service Pack for ProLiant version 2013.09.0(b)

If you are having any issues with previous drivers:

After the inbox driver, HP released a new driver version 4.0.355.1 which they then pulled as there were still issues with this driver and PXE.

HP then re-released the same driver but with a new Nic firmware requirement version 4.0.360.7. Although the numbers look very similar, 4.0.355.1 is the Nic driver version and 4.0.360.7 is the Nic firmware version.

HP then released a newer 4.0.360.20 firmware which was then pulled and replaced with 4.0.360.15a. Yes, I know .15a surely shouldn’t be released after .20!

There was however a potential known issue with the 4.0.360.15a firmware update procedure which is listed in the installation instructions:

If you receive the following error during the upgrade “Warning: One or more CRC errors was detected during the flash operation. Prior to restarting your system it is important to retry the flash update. A failed flash update may leave your system in an unknown state”

Perform the following for the adapters listed:

• BL460 or NC553i install the 2.102.517.703 iso following the above steps, reboot and reinstall the 4.0.xxx.xx.iso image
• CN1000E or NC550SFP install the 2.102.517.708 iso following the above steps, reboot and reinstall the 4.0.xxx.xx.iso image

VMware ESX/ESX(i) 4.1 Update 1/2

Gen8 / G7 Blades

You need the Emulex be2net driver version 4.1.334.0 which can be downloaded from here: http://downloads.vmware.com/d/details/dt_esxi40_emulex_be2net_403060/ZHcqYnQqaGpiZEBlZA

You need the Emulex firmware version 4.1.402.20 which can be downloaded from here: http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=lamerica_nsc_carib&prodTypeId=329290&prodSeriesId=4324629&swItem=co-106538-1&prodNameId=4324630&swEnvOID=54&swLang=8&taskId=135&mode=4&idx=1

There is a problem with Emulex firmware prior to version 4.1.450.7 that can result in SmartLink otherwise known as Device Control Channel (DCC) not working with a 10Gb physical link when you have full height blades. When you upgrade the OA, you may lose network connectivity. Update the NIC firmware and look at Customer Advisory c03600027 before upgrading the OA.

G6 Blades

HP and VMware have finally tested and certified a driver that works with DCC/SmartLink and is available through Update Manager as the shipped ESX 4.1 drivers caused PSODs.

The latest driver which supports DCC/SmartLink for ESX 4.1 is 1.70.34.v41.1

This driver can be downloaded from VMware and deployed using Update Manager.

If you are upgrading from the previously supported driver 1.60.50.v41.2 the heap memory fix is included so you don’t need to remove the workaround as was described in the original issue. http://kb.vmware.com/kb/1031805

VMware ESX/ESX i 4.0

Gen8 / G7 Blades You need the Emulex be2net driver version 4.1.334.0 which can be downloaded from here: http://downloads.vmware.com/d/details/dt_esxi40_emulex_be2net_403060/ZHcqYnQqaGpiZEBlZA

You need the Emulex firmware version 4.1.402.20 which can be downloaded from here: http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=lamerica_nsc_carib&prodTypeId=329290&prodSeriesId=4324629&swItem=co-106538-1&prodNameId=4324630&swEnvOID=54&swLang=8&taskId=135&mode=4&idx=1

G6 Blades

There is a long standing issue with the ESX Broadcom bnx2 driver. There have been quite a few drivers released, some of which support DCC/SmartLink and some that don’t.

The latest driver which supports DCC/SmartLink for ESX 4.0 is 1.62.11.v40.1

This driver can be downloaded from VMware and deployed using Update Manager.