Home > Flex-10, HP > HP Virtual Connect Flex-10 & VMware ESX(i) pre-requisites

HP Virtual Connect Flex-10 & VMware ESX(i) pre-requisites

August 9th, 2010

Updated: 28 October 2013 to include new HP September bundles and release of ESXi 5.5.

Many companies are currently going through major infrastructure changes as the reach of virtualisation into IT increases and they start moving towards a cloud model.

Blade server use is growing as people start to think more seriously about large server farms and building the infrastructure for a “cloud”, be it internal(private) or external (public). HP has been doing a lot of work selling its blades (BL460C and BL490C) in C7000 chassis with Flex-10 switches.

These infrastructure changes can be pretty complicated to implement as you need to plan for a whole new server hosting environment, not just adding another server into a rack somewhere.

In a converged world you also need to get server people, power people, VM people, storage people and networking people to work together which is a challenge at the best of times in traditional siloed enterprise IT departments.

HP has however had some problems over the years delivering the technical message to get this all done.  Unfortunately there often wasn’t enough clear information from HP about what needs to be done. Their reference architectures and Flex-10 cookbook were incomplete or out of date and in some cases contradictory and misleading. This has improved somewhat over the years but it doesn’t help that HP’s website must rate as one of the worst to find anything and it’s an IT company…quite a big one as well and it lets itself down when its hardware in my opinion is fantastic.

So, here’s a pre-requisite list I’ve been building up and maintaining over the years to hopefully get you on your way.

Onboard Administrator Firmware

HP BladeSystem c-Class Onboard Administrator (OA) Firmware

Update all your chassis OA firmware to 4.01. Although this is a v4 release the update is not nearly as dramatic as Virtual Connect 4. The Onboard Administrator upgrade from 3.71 to 4.01 just adds IPv6 support, new hardware models and a raft of bug fixes.

Update all your chassis OA firmware to 3.71.

http://h20566.www2.hp.com/portal/site/hpsc/template.PAGE/public/psi/swdDetails/?sp4ts.oid=3692952&spf_p.tpst=swdMain&spf_p.prp_swdMain=wsrp-navigationalState%3Didx%253D0%257CswItem%253DMTX_b9dd1a1e38b8457498123b9c52%257CswEnvOID%253D4064%257CitemLocale%253D%257CswLang%253D%257Cmode%253D3%257Caction%253DdriverDocument&javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken

Older version information:

There is a problem with Emulex firmware prior to version 4.1.450.7 that can result in SmartLink otherwise known as Device Control Channel (DCC) not working with a 10Gb physical link when you have full height blades. When you upgrade the OA, you may lose network connectivity. Update the NIC firmware and look at Customer Advisory c03600027 before upgrading the OA.

If you are using Gen 8 blades you will have to update your OA firmware to at least 3.50 which is the latest Recommended release as although the blades will be recognised there will be issues with Virtual Connect.

A newer version 3.55 was released the next day which fixed some authentication and EBIPA issues which must have slipped through the testing net with 3.50 (great!) . OA 3.50 can be downloaded from: http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=3709945&prodSeriesId=1844065&prodNameId=3692952&swEnvOID=4064&swLang=8&taskId=135&swItem=MTX-00b278b5e9194e7ab0ab362d55&mode=3

OA 3.55 can be downloaded from:

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=3709945&prodSeriesId=1844065&prodNameId=3692952&swEnvOID=4064&swLang=8&taskId=135&swItem=MTX-b22a6014742b4bd7a0a05557b9&mode=3

Virtual Connect Firmware

HP BladeSystem c-Class Virtual Connect Ethernet (VC-Enet), VC 4Gb FC, VC 8Gb 20-port FC, and VC 8Gb 24-port FC modules

Update your Virtual Connect Flex-10 switches to firmware 3.75 as this is a Recommended release with many bug fixes.

Virtual Connect 4.01 has been released which includes many more features and has also since been updated to 4.1 . It’s still relatively new so if you are going to use it, do your testing.

Upgrade Alert: If you have both single dense and double dense servers in your chassis and you are upgrading from Virtual Connect firmwares 3.10-3.60 up to Virtual Connect firmwares 3.70-3.75 you need to be aware of a serious issue where upgrading the firmware may cause your server profiles to be unassigned which will cause a network outage.

See this post: HP Virtual Connect Firmware upgrade may cause Server Profiles to be unassigned causing a network outage

VC 3.75 can be downloaded from: http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=3709945&prodSeriesId=3794423&prodNameId=3794431&swEnvOID=4091&swLang=8&mode=2&taskId=135&swItem=MTX-a94c23fca5ac4a21a3b88aa931

If you are using Gen 8 blades you need at least firmware version 3.51. VC 3.51 can be downloaded from: http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=3709945&prodSeriesId=3794423&prodNameId=3794431&swEnvOID=4091&swLang=8&taskId=135&swItem=MTX-bd78dea478694b249c2a637281&mode=3

If you are still using firmware pre-3.17 you need to be aware of two major issues:

HP Flex-10 issue and DNS causing network outages.

If you are using pre-3.17 firmware, see my post to implement a DNS workaround for a serious issue that can cause the Flex-10 switches to lose connectivity with each other.

HP Flex-10 3.15 firmware bug and PXE kills Virtual Connect.

If you are using 3.15 firmware, see my post about ensuring all your unused blades are powered off to avoid a bug which causes the Flex-10 switches to restart after a memory leak caused by blades continually PXE booting.

Blade Server Firmware

HP Service Pack for ProLiant

HP changed the way it does firmware and drivers and opted for an integrated approach which it is calling HP Service Pack for ProLiant (SPP) . The idea is that 3-5 times a year HP will release a whole pack of firmware and software which should all be tested together for compatibility.

You will be able to use HP Smart Update Manager (HP SUM) to look at an entire environment from HP Blades through to the Chassis OA & Virtual Connect and be able to see all interdependencies and update firmware across the whole lot in the correct order. These will now be named by the date they were released such as 2012.02.0 and hopefully always be downloadable from:

http://h18004.www1.hp.com/products/servers/service_packs/en/index.html

These SPPs will be supported for 1 year which means theoretically HP won’t force you to always upgrade to the latest drivers and firmware unless they can show that the problem you are experiencing is due to a firmware / software problem. There’s a whole bunch of documentation here which you can use to read up on it.

Unfortunately the SPP software isn’t supported for all operating systems but will be the way going forward. OS Support is for: Windows Server 2008 and later, RHEL 5 and later and SLES 10 and later.

If you are using ESXi however, you will not be actually installing any SPP software on the host but just using the firmware components so don’t get too bogged down by OS supportability.

The latest SPP is 2013.09 and can be downloaded from: http://h18004.www1.hp.com/products/servers/service_packs/en/index.html

From the 2013.02 release you can also have online updating for particular server models of System ROM, iLO, SmartArray Controller and Drive firmware with ESXi 5.x using what HP are calling Smart Components which can also use Update Manager giving you a much simpler firmware update process for the supported firmware packages.

You can see the full supported list here: http://vibsdepot.hp.com/hpq/SmartComponents/

Smart Update Firmware DVD

If you are not using ESXi or OS versions on the SPP support list you need to use the old Smart Update Firmware DVD.

The current updated version is 9.30 (2011.05.05)

Update your blade server firmware to the automatic deployment DVD version 9 or your Nics may not be visible.

Version 8.x of the firmware DVD required you to manually update components which is a pain. Boot off the DVD and it does everything automatically. Unfortunately if you run it through an iLO connection it disconnects during the update so just run it again to ensure it runs through. The host will just loop through reboot after automatic installation after reboot so once it’s been through the process the second time you are done and can disconnect the iLO DVD Drive.

You can download 9.30 (2011.05.05)  from:

http://h20566.www2.hp.com/portal/site/hpsc/template.PAGE/public/psi/swdDetails/?sp4ts.oid=3884099&spf_p.tpst=swdMain&spf_p.prp_swdMain=wsrp-navigationalState%3Didx%253D%257CswItem%253DMTX_94560dcc34e845edbb13a26792%257CswEnvOID%253D1005%257CitemLocale%253D%257CswLang%253D%257Cmode%253D%257Caction%253DdriverDocument&javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken

ESX(i) Nic Driver & Firmware Versions

Gen 8 and G7 blades use an Emulex Nic. ESXi versions up to 5.1 have the driver called  be2net. ESXi 5.5 has a  new drive model and so the Emulex drivers are now called elxnet.

G6 blades use a Broadcom bnx2x Nic for all ESXi versions.

There are two separate but related components that you need to ensure are in sync. The Nic driver software version that runs in ESXi and the Nic firmware software version which is updated separately.

HP creates ESX(i) bundles which contain supported HP software versions for various ESX(i) versions. As of Feb 2013 you can also update some firmware online from ESXi but this doesn’t include Nic firmware.

Nic Driver software can be downloaded as a .VIB file and be uploaded to Update Manager to patch your hosts or if you are using Auto Deploy you can build the driver into a custom ESXi 5.x depot using VMware Image Builder.

To manually update each host you can enable ESXi troubleshooting mode as well as SSH. Extract the .vib file and copy it to the local /tmp folder and use esxcli to install.

Example:

esxcli software vib install -v /tmp/net-be2net-4.0.355.1-1OEM.500.0.0.406165.x86_64.vib

VMware ESXi 5.5

Gen8 / G7 Blades

You need to be using Emulex elxnet nic driver version 10.0.575.7  which is included in the HP Custom Image for VMware ESXi 5.5 September 2013 or can be downloaded from VMware.

You need to be using Emulex firmware version 4.6.247.5 which can be updated using the Emulex OneConnect-Flash .ISO.

There is a problem with Emulex firmware prior to version 4.1.450.7 that can result in SmartLink otherwise known as Device Control Channel (DCC) not working with a 10Gb physical link when you have full height blades. When you upgrade the OA, you may lose network connectivity. Update the NIC firmware and look at Customer Advisory c03600027 before upgrading the OA.

G6 Blades

You need to be using Broadcom bnx2x nic driver version 1.78.28.v55.2 which is included in the HP Custom Image for VMware ESXi 5.5 September 2013 or can be downloaded from VMware.

You need to be using Broadcom firmware version 6.2.26  which can be updated using the HP Service Pack for ProLiant version 2013.09.0(b)

VMware ESXi 5.1

Gen8 / G7 Blades

You need to be using Emulex be2net nic driver version 4.6.247.10 which is included in the HP Custom Image for VMware ESXi 5.1.0 Update 1 September 2013 or can be downloaded separately from VMware.

You need to be using Emulex firmware version 4.6.247.5 which can be updated using the Emulex OneConnect-Flash .ISO.

There is a problem with Emulex firmware prior to version 4.1.450.7 that can result in SmartLink otherwise known as Device Control Channel (DCC) not working with a 10Gb physical link when you have full height blades. When you upgrade the OA, you may lose network connectivity. Update the NIC firmware and look at Customer Advisory c03600027 before upgrading the OA.

G6 Blades

You need to be using Broadcom bnx2x nic driver version 1.74.17.v50.1 which is included in the HP Custom Image for VMware ESXi 5.1.0 Update 1 September 2013 or can be downloaded separately from VMware

You need to be using Broadcom firmware version 6.2.26  which can be updated using the HP Service Pack for ProLiant version 2013.09.0(b)

VMware ESXi 5.0

Gen8 / G7 Blades

The inbox ESXi5.0 Emulex be2net driver 4.0.88.0 doesn’t work if you used VLAN tagging anywhere in your network path and also has issues with PXE which will affect you if you wanted to use Auto Deploy.

You need to be using Emulex be2net nic driver version 4.6.247.07 which is included in the HP Custom Image for VMware ESXi 5.0.0 Update 3 October 2013 or can be downloaded separately from VMware.

You need to be using Emulex firmware version 4.6.247.5 which can be updated using the Emulex OneConnect-Flash .ISO.

There is a problem with Emulex firmware prior to version 4.1.450.7 that can result in SmartLink otherwise known as Device Control Channel (DCC) not working with a 10Gb physical link when you have full height blades. When you upgrade the OA, you may lose network connectivity. Update the NIC firmware and look at Customer Advisory c03600027 before upgrading the OA.

G6 Blades

You need to be using Broadcom bnx2x nic driver version 4.2.327.0 which is included in the HP Custom Image for VMware ESXi 5.0.0 Update 2 February 2013VMware-ESXi-5.0.0-Update2-914586-HP-5.27.20.iso or can be downloaded separately from VMware

You need to be using Broadcom firmware version 6.2.26  which can be updated using the HP Service Pack for ProLiant version 2013.09.0(b)

If you are having any issues with previous drivers:

After the inbox driver, HP released a new driver version 4.0.355.1 which they then pulled as there were still issues with this driver and PXE.

HP then re-released the same driver but with a new Nic firmware requirement version 4.0.360.7. Although the numbers look very similar, 4.0.355.1 is the Nic driver version and 4.0.360.7 is the Nic firmware version.

HP then released a newer 4.0.360.20 firmware which was then pulled and replaced with 4.0.360.15a. Yes, I know .15a surely shouldn’t be released after .20!

There was however a potential known issue with the 4.0.360.15a firmware update procedure which is listed in the installation instructions:

If you receive the following error during the upgrade “Warning: One or more CRC errors was detected during the flash operation. Prior to restarting your system it is important to retry the flash update. A failed flash update may leave your system in an unknown state”

Perform the following for the adapters listed:

  • BL460 or NC553i install the 2.102.517.703 iso following the above steps, reboot and reinstall the 4.0.xxx.xx.iso image
  • CN1000E or NC550SFP install the 2.102.517.708 iso following the above steps, reboot and reinstall the 4.0.xxx.xx.iso image

VMware ESX/ESX(i) 4.1 Update 1/2

Gen8 / G7 Blades

You need the Emulex be2net driver version 4.1.334.0 which can be downloaded from here: http://downloads.vmware.com/d/details/dt_esxi40_emulex_be2net_403060/ZHcqYnQqaGpiZEBlZA

You need the Emulex firmware version 4.1.402.20 which can be downloaded from here: http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=lamerica_nsc_carib&prodTypeId=329290&prodSeriesId=4324629&swItem=co-106538-1&prodNameId=4324630&swEnvOID=54&swLang=8&taskId=135&mode=4&idx=1

There is a problem with Emulex firmware prior to version 4.1.450.7 that can result in SmartLink otherwise known as Device Control Channel (DCC) not working with a 10Gb physical link when you have full height blades. When you upgrade the OA, you may lose network connectivity. Update the NIC firmware and look at Customer Advisory c03600027 before upgrading the OA.

G6 Blades

HP and VMware have finally tested and certified a driver that works with DCC/SmartLink and is available through Update Manager as the shipped ESX 4.1 drivers caused PSODs.

The latest driver which supports DCC/SmartLink for ESX 4.1 is 1.70.34.v41.1

This driver can be downloaded from VMware and deployed using Update Manager.

If you are upgrading from the previously supported driver 1.60.50.v41.2 the heap memory fix is included so you don’t need to remove the workaround as was described in the original issue. http://kb.vmware.com/kb/1031805

VMware ESX/ESX i 4.0

Gen8 / G7 Blades You need the Emulex be2net driver version 4.1.334.0 which can be downloaded from here: http://downloads.vmware.com/d/details/dt_esxi40_emulex_be2net_403060/ZHcqYnQqaGpiZEBlZA

You need the Emulex firmware version 4.1.402.20 which can be downloaded from here: http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=lamerica_nsc_carib&prodTypeId=329290&prodSeriesId=4324629&swItem=co-106538-1&prodNameId=4324630&swEnvOID=54&swLang=8&taskId=135&mode=4&idx=1

G6 Blades

There is a long standing issue with the ESX Broadcom bnx2 driver. There have been quite a few drivers released, some of which support DCC/SmartLink and some that don’t.

The latest driver which supports DCC/SmartLink for ESX 4.0 is 1.62.11.v40.1

This driver can be downloaded from VMware and deployed using Update Manager.

Categories: Flex-10, HP Tags: , , , ,
  1. August 9th, 2010 at 22:57 | #1

    Hey Julian:

    I would love to see what we at HP could do to make your life easier. Can you give me details on what you did/did not find, like, etc. on the BladeSystem info you were looking for and using. Maybe I can help and in doing so help others using BladeSystem. We’re not that proud that we can’t improve what you experienced and found lacking. You have my email address.

    Thanks.

  2. beb4vm
    August 24th, 2010 at 18:15 | #2

    There are still issues with configuration choices (namely if you want DCC functionality) and which version of ESX you want to run. You can read all of the threads I have been posting on here:

    http://communities.vmware.com/thread/273033?

    http://h30499.www3.hp.com/t5/HP-BladeSystem-Virtual-Connect/Understanding-link-state-DCC-issues-with-ESX4-and-Virtual/m-p/1569/highlight/false

    And here is HP’s most recent Customer Advisory:
    http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&objectID=c02476622&jumpid=reg_R1002_USEN

  3. February 11th, 2011 at 17:43 | #3

    Hi Julian,

    Will the BL490c G6 and G7 show a hardware iscsi adapter in esxi 4.1 with all the correct firmwares and drivers?

    I have OA 2.13 VC 3.15 and the newest set of bnx2 1.6 on the esxi 4.1 g7 blade.

    I can only ever get software iscsi adapters?

  4. Phil Brown
  5. WoodITWork
    March 22nd, 2011 at 10:41 | #5

    @Phil Brown
    Thanks for the updated link, it is very annoying that HP don’t keep persistent links for their downloads.

  6. June 14th, 2011 at 07:40 | #6

    Julian, thanks for all your posts about VC and Vmware. That’s exactly what I needed to start designing our new infrastructure. HP docs are so boring.

    Short note regarding firmware upgrade – we need to upgrade our C7K with Flex-10 VC and HP engineers recommendations were to upgrade it in the following order: OA, ILO, Blades and then only VC. Another recommendation was to upgrade to the latest release set 9.30, but in your post I see that you advise to upgrade blades to 9.20 (B). Is there any specific reason for this? The last advise from HP was to use VC Support Utility 1.52 for VC Ethernet module, and VCSU 1.51 for VC FC module as they had some issues with VCSU and FC lately. According to them these issues will be fixed in 1.53

  7. December 16th, 2011 at 13:09 | #7

    We have some ProLiant BL490c G7 blades.
    Bye the way, Thanks for the brilliant info here Julian!!!
    I finally found the Emulex firmware at this location.
    ftp://ftp.hp.com/pub/softlib/software12/COL35717/co-99913-1/OneConnect-Flash-4.0.360.20.iso
    I just cannot believe how slow this download is, and that towards a professional audience!!!!
    Anyway, the install tool is 4.0.360.20, firmware is really 4.0.360.15.
    For me this still gives tagging issues with ESXi 5.0 (with Vmware 4.0.355.1 driver installed)
    It is a real pain in the neck.
    There is such a big market for blades with ESXi 5.0, how can these issues be there.

  8. December 19th, 2011 at 14:06 | #8

    @Peter Verijke
    Tagging problem is solved with the new firmware/esxi driver.
    The problem was related to the host itself (Checkpoint Firewall)
    Everything is running smooth now.

  9. Kieran
    December 21st, 2011 at 15:53 | #9

    Hi!
    I still seem to be having this problem after the FW update. only when I am running more than 3 blades at a time…!?
    Does anyone have any more info on this?!

  10. Rotem Agmon
    January 27th, 2012 at 19:02 | #10

    Hi Julian,

    Emulex firmware version 4.0.360.20 iso image is no longer available for download and has been replaced with version 4.0.360.15a

    Please update the download link:

    http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=lamerica_nsc_carib&prodTypeId=329290&prodSeriesId=4324629&prodNameId=4324630&swEnvOID=54&swLang=8&mode=2&taskId=135&swItem=co-101629-1

  11. WoodITWork
    January 28th, 2012 at 14:00 | #11

    @Rotem Agmon

    Hi Rotem, thanks for the update.

    Are you aware of the issue that is described in this post:
    http://www.virtualinsanity.com/index.php/2012/01/27/hp-emulex-cna-firmware-problems-on-bl460g7/

  12. Rotem Agmon
    January 30th, 2012 at 13:40 | #12

    @WoodITWork

    Hi Julian,

    Yes I’ve seen this issue and tried commenting on the linked blog post but no avail.

    The issue described is a known issue with the latest Emulex firmware update, see installation instructions for a solution:

    Important: If you receive the following error during the upgrade ”Warning: One or more CRC errors was detected during the flash operation. Prior to restarting your system it is important to retry the flash update. A failed flash update may leave your system in an unknown state. ”

    Perform the following for the adapters listed:

    – BL460 or NC553i install the 2.102.517.703 iso following the above steps, reboot and reinstall the 4.0.xxx.xx.iso image

    http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=lamerica_nsc_carib&prodTypeId=329290&prodSeriesId=4324629&swItem=co-101629-1&prodNameId=4324630&swEnvOID=54&swLang=8&taskId=135&mode=4&idx=0

  13. Rotem Agmon
    February 7th, 2012 at 21:02 | #13

    Remember that if you are using your NC553i CNA Adapter for FCoE/FC connectivity (with FlexFabric) you need to install Emulex lpfc driver version 8.2.2.105.36

    if you use the nic for iSCSI connectivity you also need to install the Emulex be2iscsi driver version 4.0.317.0

    For download link follow:
    http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2007397

  14. Björn
    February 13th, 2012 at 13:56 | #14

    Thanks for a good guide.

    Please note that VMware also has an Support Matrix for what firmware versions is supported with ESXi x.x

    http://partnerweb.vmware.com/programs/hcl/ESX_Flex_config.pdf

    Also, what I could understand Smart Update Firmware DVD 9.30 (2011.05.05) was the last one HP released. The new black is the “HP Service Pack for ProLiant” http://www.hp.com/go/spp

  15. March 12th, 2012 at 15:04 | #15

    Great post…

  16. Rotem Agmon
    March 13th, 2012 at 19:25 | #16

    @Rotem Agmon
    Update:

    Emulex firmware version 4.0.360.15a iso image is no longer available for download and has been replaced with version 4.0.360.15b

    Please update the download link:
    http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=3709945&prodSeriesId=4194735&prodNameId=4194638&swEnvOID=54&swLang=8&mode=2&taskId=135&swItem=co-102634-1&prodTypeId=329290&prodSeriesId=4111422

  17. April 23rd, 2012 at 07:48 | #17

    I am wondering how different are VC FlexFabric prerequisites for G7 blades with ESXi 5 Update 1?

    Yesterday failed to upgrade HP Blade enclosure to the latest firmware. Upgrade of OA, ILO, Blades went fine. OA is 3.55 now, ILO – 1.28, Blades Proliant 460c G7 were updated with SmartUpdate 10.0
    .
    Some additional firmware/driver version info

    ~ # cat /proc/scsi/lpfc820/0
    Emulex OneConnect FCoE SCSI 8.2.2.105.36
    Firmware Version: 4.0.493.0

    ~ # ethtool -i vmnic0
    driver: be2net
    version: 4.0.355.1
    firmware-version: 4.0.493.0

    However, when I updated VC FlexFabric to 3.51 my datastore latency I could see in ESXi5 went really bad, I could see spikes of 20-30s!, not ms.
    Reset of the VC FlexFabric modules didn’t help. Neither did the total power shutdown of the entire enclosure. So, I had to downgrade the VC FlexFabric firmware back to 3.18, which finally helped to resolve the problem – thanks a lot for posting the procedure on how to do it, really appreciate it.

    will have another enclosure in 1 month so would be able to test it again.

  18. Rabie van der Merwe
    May 2nd, 2012 at 08:59 | #18

    It seems that both HP and VMware is recommending updated drivers for ESXi 4.1 and G7 blades, 4.0.306.0:
    http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2007397

    Also HP seems to have updated their ESXi 4.1u2 installer images to contain the latest said drivers.

  19. Rotem Agmon
    June 10th, 2012 at 04:45 | #19

    Please use the newly released “June 2012 VMware FW and Software Recipe” for recommended VMware drivers and firmware versions for all HP servers:

    http://vibsdepot.hp.com/hpq/recipes/June2012VMwareRecipe1.0.pdf

  20. Håkan
    June 14th, 2012 at 09:02 | #20

    @Rotem Agmon
    According to the pdf the recommended firmware for be2net is 4.1.402.8. Do you know where I can find it?

  21. Rotem Agmon
    June 15th, 2012 at 16:19 | #21

    4.1.402.8 firmware is included with HP Service Pack for ProLiant 2012.06.0(B) which has been released and available for download:

    http://h18004.www1.hp.com/products/servers/service_packs/en/index.html

    The firmware hasn’t yet become available as a separate download.

  22. Rabie van der Merwe
    June 21st, 2012 at 05:45 | #22

    It’s still extremely anoying that neither HP nor VMware makes any proper changelogs availible on the updated drivers so you have little to no idea if it fixed something or really just addes new features/bugs. However I have managed to find what the updated firmware fixes:
    http://www-dl.emulex.com/support/hardware/oce10102/fw/4140218/fw_bootcode_release_notes.pdf

  23. DaveK
    June 24th, 2012 at 07:57 | #23

    These driver issues continue in my environment. We reacted to the following:
    http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2010222
    and pushed out the latest be2net driver to one of our clusters. Now we experience random disconnects of either the vm management network NICs (a standard vSwitch) or the NICs associated with the virtual machine dvSwitch.
    Here’s the driver version I’m referring to – 4.1.334.48
    Our main production cluster still hasn’t received the driver, and we are not seeing the random disconnects. We are, however, experiencing random reboots on those blades. This may be a firmware issue.
    I’ve spent the better part of my time in the last 6 to 8 months chasing these issues. More than a little annoying. At any rate, thought I’d advise anyone willing to listen that you might want to hold off on that latest driver. I can’t say with certainty that its the cause of the disconnects, but it has not helped.

  24. Rabie van der Merwe
    June 25th, 2012 at 06:55 | #24

    I cant really talk about ESXi5.0 but I see that with the latest HP SPP (2012_06B) they have introduced new firmware (listed above) and new drivers (4.1.402.8 and 4.1.334.0 respectively) they have also updated their install ISO to use said drivers. Although I have not seen any reboots or disconnects per say, what I have seen on the previous driver is that 1 node cannot ping another node yet the second node can ping the first node and externally both can be pinged which causes HA to complain and have issues. I have updated to the latest firmware and driver but are still on 3.18 on VC. Will see how it goes.

  25. Rotem Agmon
    June 30th, 2012 at 20:58 | #25

    4.1.402.8 firmware that is included with HP Service Pack for ProLiant 2012.06.0(B) has been superseded by version 4.1.402.20 which is available for download as a standalone image:
    http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=lamerica_nsc_carib&prodTypeId=329290&prodSeriesId=4324629&swItem=co-106538-1&prodNameId=4324630&swEnvOID=54&swLang=8&taskId=135&mode=4&idx=1

  26. Nathaneal
    July 5th, 2012 at 09:50 | #26

    Many thanks for you information this has been very helpful. I have one issue and we cannot get to the bottom of it, we are currently monitoring the system with Solarwinds and the ESXi hosts show hundreds of Millions of discards.

    We cannot figure this out, we have contacted HP but it is not a network problem as a full investigation and the VC or the switches show no discards. Its only the vmnic 0, we have the same configuration as your documents, as we based our systems on your design, and run a active/active configutation, it works a treat but we cannot understand the discards.

    As mentioned we have latest patches, firmware, drivers.

  27. Josef
    July 8th, 2012 at 04:01 | #27

    Thank you for keeping this updated with the latest information.

    We have an issue with regards to our ESXi 5.0 servers (BL460c G7) loosing sight of our IBM SAN’s LUNs after a controlled shutdown.

    fingers crossed these new firmware and drivers fix this.
    Virtual Connect (3.60)
    Onboard Administrator A (3.56)
    emulex firmware be2net 4.1.402.8, drivers be2net-4.1.334.0-623960

  28. Nathaneal
    July 9th, 2012 at 09:20 | #28

    Hello, can you keep us posed with this as the only thing not updated is the firmware of the Virtual Connect and Onboard Administrator, we are on the previous versions.

  29. Nathaneal
    July 12th, 2012 at 15:46 | #29

    After talking and working with vmware we found that the discards problem is down to the emulex driver, they have submitted a case to Emulex to inform them of our findings.

    We are showing no discards with this driver on our vmware nic’s now and this has been running for two days. The others servers with the latest Emulex driver and not the one vmware gave us are still showing hundreds of millions of discards.

    Hopefully emulex will send out a update soon so we can patch all our systems and keep solarwinds tidy.

  30. Rabie van der Merwe
    July 16th, 2012 at 13:30 | #30

    Hi Guys,
    Found an HP advisory that suggestes that this only affects servers with FCoE and DMA conflicts on Gen8, the document is a little misleading on whether the issue is only on Gen8 or whether it also affects people running Multipath FCoE as well. Hp Advisory c03400156
    http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c03400156&jumpid=em_alerts_us-us_Jul12_xbu_all_all_1808520_119812_bladesystemoptionsandaccessoriesstorage_critical_013_4

  31. Richard
    August 22nd, 2012 at 13:07 | #31

    Hi Nathaneal,
    Your issue seems to exactly match what we are getting (even down to our Solarwinds getting noisy). I’d like to log a call with VMWare, but before I do this, do you have any other information – am anxious not to waste an afternoon (which you have probably done already) getting them to understand the problem when quoting a call reference or something like that would save time.
    Help appreciated.
    Richard

  32. Nathaneal
    August 22nd, 2012 at 14:47 | #32

    Hi,

    Yep, vmware have reported to Emulex and we are currently waiting on them to release a new driver to solve the issue.

    This should get rid of all the discards as testing with vmware, we installed a debug driver and they all disappeared.

    Vmware report to Emulex, so just waiting now.

    Thanks

  33. Rabie van der Merwe
    September 10th, 2012 at 07:41 | #33

    Hi,

    It looks like HP has been busy again and finding release notes is almost impossible. They have release another support pack 2012_08 which contains updated OA and VC (if you have BL2x220 DOT NOT USE OA 3.60, there is a 3.61 with a date prior to 3.60 but it is newer) and they have release 3.70 which resolved the upgrade bug and LAG timing issue, they have also put out a new VSCU utilitly 1.70. Then they have ALSO upped the firmware on the BE2 and BE3 cards to 4.1.450.7, up from 4.1.402.8 and 4.1.402.20. I have managed to find some release notes on that:
    Added support for the following operating systems: 1. RHEL 6.3
    Resolved Issues from Firmware Version 4.1.402.20:
    1. SR IOV: Unable to ping IPV6 from DUT VF to Peer VF
    2. Running ethtool with BE3 takes longer than expected after cable pull
    3. FAT_CFG_SNAPSHOT command timed out
    4. Out of Order CQEs
    5. Data integrity Issue noted during short duration cable pulls
    6. W2K8 graceful shutdown failed under B120i with NC552
    7. BL460c Gen8 with 554FLB Unable to PXE boot
    8. 1Gb LOM ports in Windows do not come back up when switch is power cycled
    9. RSCN is not generated by switch when a driver restarts
    10. 7 second delay in GET_CARD_ATTRIBUTES command
    11. BE3 read sfp ioctl returns bad data on twinax cables that BE2 returns good data with
    12. 554FLR FCoE Path Multipath Recovery Fail SLES11 sp1 x64
    13. Out of order Abort completion and IO completion CQE
    14. Received CQE_EXT_INVALID_RPI in abort completion of ELS command
    15. Correction to OCSD to prevent DMA conflicts with LPFC driver
    (http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=3709945&prodSeriesId=4132949&swItem=co-108046-1&prodNameId=4132827&swEnvOID=54&swLang=8&taskId=135&mode=4&idx=0)

  34. Rotem Agmon
  35. Håkan
    October 10th, 2012 at 10:28 | #35

    Here we go again. I was just about to upgrade our enclosures and G7-blades to 2012.08 SPP (G8-blades ordered..) with G7 nic fw 4.1.450.7 but then I saw this one and since we have lots of G7 ESXi 5.0 hosts I guess I should go with this one after reading the resolved issues section:

    4.1.450.16 (8 Oct 2012)

    Resolved Issues from Firmware Version 4.1.450.7

    BladeEngine Firmware

    1. HP P9500 crashes with 4.1.402.x, 4.1.450.7 firmware
    2. EFI: By default same iSCSI name is set to both the ports of an HBA
    3. Fixed a VMware ESXi 5.0 PSOD
    4. Changed SERDES settings for 3m/3.5m DAC cables

    PXE BIOS

    1. PXE Boot fails on reboot with CN1000E

    UEFI iSCSI

    1. Added Auto Update feature
    2. Fixed BIOS Version displayed in controller properties in EFI

    Known Issues

    There are no known issues for this release.

    http://h20566.www2.hp.com/portal/site/hpsc/template.PAGE/public/psi/swdDetails/?sp4ts.oid=4194638&spf_p.tpst=psiSwdMain&spf_p.prp_psiSwdMain=wsrp-navigationalState%3Dlang%253Den%257Ccc%253DUS%257CprodSeriesId%253D4194735%257CprodNameId%253D4194638%257CswEnvOID%253D54%257CswLang%253D8%257CswItem%253Dco-108976-1%257Caction%253DdriverDocument

  36. April 3rd, 2013 at 19:56 | #36

    Just wanted to share my experiance, it might help someone out there struggling with BL460c G6 blades running vSphere 5.0U2 on Virtual Connect and Flex-10. Having installed January 2013 firmware and trying to use February release of HP branded vSphere media we could not get our scripted install to function consistently (it would only work 20% of the time – how frustrating is that!). After weeks of struggling we used VMWare’s suggestion to just use their non-branded media; problem solved! We just had to add some HP specific bundles to the script and we’re on our way. Having lothe HP software for years I’m not sure why I did not try it first. These people don’t test anything, bugs fixed month ago usually re-appear on next or after realease.

  37. michael schweiger
    June 4th, 2013 at 13:09 | #37

    And again, just couple of seconds ago the FW got replaced…

    http://h20566.www2.hp.com/portal/site/hpsc/template.PAGE/public/psi/swdDetails/?sp4ts.oid=4132830&spf_p.tpst=swdMain&spf_p.prp_swdMain=wsrp-navigationalState%3Didx%253D1%257CswItem%253Dco_117281_1%257CswEnvOID%253D54%257CitemLocale%253D%257CswLang%253D%257Cmode%253D4%257Caction%253DdriverDocument&javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken

    OneConnect UCNA Firmware Release Notes for HP, April 2013
    Firmware Version 4.2.401.2215
    BE2-based Models:
    NC550SFP
    NC550m
    CN1000E
    NC551m
    NC551l BL465C-G7
    NC551l BL685C-G7
    BE3-based Models:
    NC553l BL460c G7
    NC553l BL490c G7
    NC553l BL620c G7
    NC553l BL680c G7
    NC553m
    NC552m
    NC552SFP
    CN1100E
    554FLR-SFP+
    554FLB
    554M
    552M
    Purpose and Contact Information
    These release notes describe the content, resolved known issues, and
    current known issues associated with this OneConnect™ Universal
    Converged Network Adapter (UCNA) firmware/boot code release.
    Content:
    Resolved Issues from Firmware Version 4.2.401.605
    BladeEngine Firmware

    1. Fixed VMware PSOD ESX 4.1/5.0/5.1 seen on BL465cG7, BL685cg7
    servers with BE2 based cards
    2. Fixed FW memory leak with multiple logins to a redirected target
    3. Fixed FW to allow setting Speed/Duplex to Auto/Auto
    4. Fixed Fw to allow Bonding with Citrix platforms
    5. Fixed FW to allow upgrades of version 4.x to 4.x+ without requiring cold
    boot
    6. Fixed UEFI to not require 2nd reset
    7. Fixed UEFI driver version reported
    8. Fixed FW to prevent LOMS at 10G from negotiating 1G with Blade switches
    9. Fixed FW for BE3 to report correct values of ethernet stat “In Range” error
    10. Fixed FW to boot from secondary iSCSI when selected
    11. Fixed FW iSCSI boot failure from secondary target when primary target given
    is incorrect
    12. Fixed UEFI NIC default settings retention
    13. Fixed FW hang during boot with SLES 10 SP4
    14. Fixed FW error while performing Shared Uplink Set port remove/add test

    Known Issues
    1. Firmware 4.2.401.6 is not compatible with Rhel 6.2×64 Inbox NIC driver
    (version 4.1.255.11). In order to upgrade to Firmware 4.2.401.6, you must first
    upgrade to Rhel 6.2×64 driver version 4.2.392.1 or later.
    Symptom seen with 4.1.255.11 inbox driver: “TX hang in standard error as well as
    /var/log/messages”

  38. June 4th, 2013 at 16:57 | #38

    @michael schweiger

    Please note that the latest firmware release that is currently compatible with vSphere 5.x and HP Virtual Connect/FlexFabric is version 4.2.401.605

  39. Dave Kinsley
    June 4th, 2013 at 18:28 | #39

    Actually, HP just released new firmware – 4.2.401.2215. There is an accompanying be2net driver for ESXi 5.x – 4.2.327.9

  40. Michael Schweiger
    June 5th, 2013 at 08:52 | #40

    @Rotem:

    and today this advisory was posted (for bl685c G7, and other “big machines”)

    SUPPORT COMMUNICATION -CUSTOMER ADVISORY
    Document ID: c03787760
    Version: 1
    Advisory: Certain ProLiant Servers Running VMware -FIRMWARE UPDATE REQUIRED: Servers May Unexpectedly Reset, Experience a Machine Check, PCI Bus Error, or Purple Screen of Death (PSOD)
    NOTICE:The information in this document, including products and software versions, is current as of the Release Date. This document is subject to change without notice.
    Release Date: 2013-06-04
    Last Updated: 2013-06-04
    DESCRIPTION
    IMPORTANT: The Firmware update detailed below is considered a critical fix and is required to prevent the issues detailed below. HP strongly recommends immediate application of this critical fix. Neglecting to perform the required action could leave the server in an unstable condition, which could potentially result in sub-optimal server performance, server lock-ups or failures, or ungraceful server shutdowns. By disregarding this notification and not performing the recommended resolution, the customer accepts the risk of incurring future related errors.
    On the ProLiant servers detailed in the Scope section below, any or all of the following errors may occur if the server is configured with VMware ESXi 4.x and 5.x and either of the adapters outlined below:
    Machine Check Error

    PCI Bus Error

    The active network may appear to act as if it is disconnected without passing any traffic but still show a hardware link

    Purple Screen of Death (PSOD)

    Affected Adapters:
    HP NC551i network adapter integrated in the ProLiant BL685c G7 server

    HP NC550SFP 10Gb 2-port PCIe x8 Ethernet Adapter configured in a ProLiant DL585 G6/G7 or ProLiant DL785 G6 server

    SCOPE
    Any ProLiant BL685c G7 server blade with an onboard NC551i LOM or any ProLiant DL585G6, ProLiant DL585 G7, or ProLiant DL785 G6 server configured with an optional NC550SFP Gigabit Server Adapter and running VMware ESX/ESXi 4.1, 5.0, and 5.1, including all Update levels.
    IMPORTANT: This issue has only been observed in a VMware environment. Other operating systems are not affected.
    RESOLUTION
    Update the Emulex Adapter firmware to version 4.1.450.1707 or 4.2.401.2215. Ensure that the matching driver family version, 4.1.x.x or 4.2.x.x, is also used with the updated firmware version:
    4.1.450.1707 is located here:
    http://h20566.www2.hp.com/portal/site/hpsc/template.PAGE/public/psi/swdDetails/?sp4ts.oid=4132830&spf_p.tpst=swdMain&spf_p.prp_swdMain=wsrp-navigationalState=idx%3D%7CswItem%3Dco_116706_1%7CswEnvOID%3D54%7CitemLocale%3D%7CswLang%3D%7Cmode%3D%7Caction%3DdriverDocument&javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken
    4.2.401.2215 is located here:
    ftp://em050713:^DM7fu$F@ftp.usa.hp.com/NIC Firmware/BE/OneConnect-Flash-4.2.401.2215.iso
    RECEIVE PROACTIVE UPDATES: Receive support alerts (such as Customer Advisories), as well as updates on drivers, software, firmware, and customer replaceable components, proactively via e-mail through HP Subscriber’s Choice. Sign up for Subscriber’s Choice at the following URL: http://www.hp.com/go/myadvisory
    NAVIGATION TIP: For hints on navigating HP.com to locate the latest drivers, patches, and other support software downloads for ProLiant servers and Options, refer to the Navigation Tips document.
    SEARCH TIP: For hints on locating similar documents on HP.com, refer to the Search Tips document.
    Hardware Platforms Affected: HP NC550SFP 10Gb 2-port PCIe x8 Ethernet Adapter(Standard HP Product), HP ProLiant BL685c G7 Server Blade, HP ProLiant DL585 G6 Server, HP ProLiant DL585 G7 Server, HP ProLiant DL785 G6 Server
    c03787760
    Mittwoch, 5. Juni 2013
    09:31
    Emulex CNA Page 1
    DL785 G6 Server
    Operating Systems Affected: VMware ESXi 4.1(Standard HP Product), VMware ESXi 5.0(Standard HP Product), VMware ESXi 5.1(Standard HP Product)
    Software Affected: Not Applicable
    Support Communication Cross Reference ID: IA03787760
    ©Copyright 2013 Hewlett-Packard Development Company, L.P.
    Hewlett-Packard Company shall not be liable for technical or editorial errors or omissions contained herein. The information provided is provided “as is” without warranty of any kind. To the extent permitted by law, neither HP or its affiliates, subcontractors or suppliers will be liable for incidental,special or consequential damages including downtime cost; lost profits;damages relating to the procurement of substitute products or services; or damages for loss of data, or software restoration. The information in this document is subject to change without notice. Hewlett-Packard Company and the names of Hewlett-Packard products referenced herein are trademarks of Hewlett-Packard Company in the United States and other countries. Other product and company names mentioned herein may be trademarks of their respective owners.
    Emulex CNA Page 2

  41. albertwt
    September 17th, 2013 at 15:14 | #41

    Hi people,

    Do I need to upgrade my HP blade BL 490c G7 Firmware before installing the HP Custom Image for VMware ESXi 5.1.0 Update 1 Install CD from VMware website or from the https://vibsdepot.hp.com/hpq/custom/vsphere51U1 site ?

    • Julian Wood
      September 23rd, 2013 at 14:54 | #42

      Its always best to ensure your blade firmware is up to date before installing any software. Order should be Blade Firmware, VC Firmware, OS/Hypervisor software drivers

  42. albertwt
    September 23rd, 2013 at 23:12 | #43

    Julian, many thanks for the clarification, so in this case for the sake of installing ESXi 5.1 Update 1, I can just upgrade the HP Emulex 10GbE Converged Network Adapters and 10GbE Network Adapters Firmware – Network manually by using the OneConnect-Flash-4.6.247.5.iso (35 MB)

    after that if no issues arise (hopefully), only upgrade the Blade Enclosure, Blade Firmware and then Virtual Connect firmware with SPP DVD.

    How to prevent major outage for all of the VMs when upgrading those firmware for the Blade Enclosure (c7000 chassis) and the Virtual Connects ?

  43. Marco
    November 11th, 2013 at 09:43 | #44

    “You need to be using Broadcom firmware version 6.2.26 which can be updated using the HP Service Pack for ProLiant version 2013.09.0(b)”

    Unfortunately this firmware doesn’t come up in HPSUM where updating a G6 blade with a NC532m NIC. Also, I can’t seem to find it anywhere (HP support center & software depot, Broadcom). Did someone find this particular firmware? Thanks.

  44. Bjorn
    November 16th, 2013 at 02:07 | #45

    @Marco

    Same thing for me. My G7 servers did get a new firmware from 2013.09.0(b), no need for OneConnect iso. But G6 firmware is nowhere to be found. If I google it, most hits points to here.

    Firmware for ESXi 5.5 should be 7.8.xx according to:
    http://partnerweb.vmware.com/programs/hcl/ESX_Flex_config.pdf

    Cant find that one either.

  45. Bjorn
    November 16th, 2013 at 16:45 | #46

    @Bjorn
    I found the NIC firmware G6 (at least BL460c G6), it’s in the “HP Broadcom Online Firmware Upgrade Utility for VMware 5.x”, search for BL460c G6 on support site and it’s there. The file you want and is named CP021532.scexe

    However, I failed to deploy the CP021532.scexe via the fantastic HP SUM [/sarcasm]. This is how I did:

    1) SCP CP021532.scexe to ESXi host /tmp
    2) cd tmp
    3) chmod +x CP021532.scexe
    4) ./CP021532.scexe
    5) Answer on the questions that appear and reboot.
    6) Login after reboot and run: ethtool -i vmnic0
    7) You’ll have firmware 6.2.26

    Boom!

    /Björn

  46. Marco
    November 20th, 2013 at 06:54 | #47

    @Bjorn
    Got it. Thanks a lot.

    Also thanks to Julian for keeping this great article up to date!

  47. Bjorn
    November 20th, 2013 at 09:21 | #48

    Marco :
    @Bjorn
    Got it. Thanks a lot.
    Also thanks to Julian for keeping this great article up to date!

    No problem! I have now upgraded OA and all blade servers, no issues detected. “Only” virtual connect modules (3.70 -> 4.10) left… 🙂

    As Marco wrote, totally agree. Thank you Julian for all your great posts! Wooditwork.com is imo the best source for information on HP Blade/VMware. Keep up the good work!

  48. Michael Schweiger
    August 6th, 2014 at 09:31 | #49

    And once again troubles with G7 and Gen8 servers and Emulex (probably other vendors as well)

    Emulex OneConnect network cards missing with elxnet driver 10.0.725.2 and later in ESXi 5.5 (2076072)

    To resolve this issue, use one of these options:
    Remain on the inbox elxnet driver 10.0.575.7 which comes with ESXi 5.5
    Install an updated driver but disable the NetQueue/MSIX functionality which will allow all 8 network cards to be detected. Run the below command to disable NetQueue/MSIX functionality

    esxcfg-module -s “msix=0” elxnet

    http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2076072

  1. No trackbacks yet.
Comments are closed.