HP Blade Emulex NC55X Nic firmware may cause network disconnections
HP has released an advisory saying that with versions of the Emulex Nic driver firmware pre 4.1.450.7 Device Control Channel (DCC) may not work which could cause your blades with NC55x Nics to drop off the network if the upstream network connection fails. This has been an issue which has plagued HP blade networking since the beginning and was a particular problem with the G6 Broadcom Nics so its disconcerting to see similar issues resurface.
4.1.450.7 is older firmware however and my post HP Virtual Connect Flex-10 & VMware ESX(i) pre-requisites recommends Emulex firmware version 4.2.401.605 so if you haven’t updated, now is the time.
HP blade servers have hard wired connections to their immediate upstream Virtual Connect switch via the chassis backplane. This means in normal operation the Nic on the blade will never see a down state unless the physical Virtual Connect switch is powered off or removed. This would still be the case even if the upstream network connection failed if it wasn’t for DCC. DCC is a firmware component where Virtual Connect sees all uplinks for an Ethernet Network are down and then sends a command down to the individual Flex Nics to tell them to mark the port as down so the blade can fail over traffic to another nic in a port group or team. DCC is what SmartLink does and requires coordination between Virtual Connect firmware and the blade Nic firmware which in the past has caused a lot of issues.
Something I noticed in the advisory however was you can actually now see whether DCC/SmartLink is working for each individual network connection. Navigate to the Interconnect Bays in the VCM GUI, then navigate to the server ports and view the DCC state:
{Source: HP}
This looks like a recent addition to Virtual Connect firmware and will certainly help troubleshooting network connection issues but hopefully the DCC disconnect debacle won’t be repeated.
And again, just couple of seconds ago the FW got replaced…
http://h20566.www2.hp.com/portal/site/hpsc/template.PAGE/public/psi/swdDetails/?sp4ts.oid=4132830&spf_p.tpst=swdMain&spf_p.prp_swdMain=wsrp-navigationalState=idx%3D1%7CswItem%3Dco_117281_1%7CswEnvOID%3D54%7CitemLocale%3D%7CswLang%3D%7Cmode%3D4%7Caction%3DdriverDocument&javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken
OneConnect UCNA Firmware Release Notes for HP, April 2013
Firmware Version 4.2.401.2215
BE2-based Models:
NC550SFP
NC550m
CN1000E
NC551m
NC551l BL465C-G7
NC551l BL685C-G7
BE3-based Models:
NC553l BL460c G7
NC553l BL490c G7
NC553l BL620c G7
NC553l BL680c G7
NC553m
NC552m
NC552SFP
CN1100E
554FLR-SFP+
554FLB
554M
552M
Purpose and Contact Information
These release notes describe the content, resolved known issues, and
current known issues associated with this OneConnect™ Universal
Converged Network Adapter (UCNA) firmware/boot code release.
Content:
Resolved Issues from Firmware Version 4.2.401.605
BladeEngine Firmware
1. Fixed VMware PSOD ESX 4.1/5.0/5.1 seen on BL465cG7, BL685cg7
servers with BE2 based cards
2. Fixed FW memory leak with multiple logins to a redirected target
3. Fixed FW to allow setting Speed/Duplex to Auto/Auto
4. Fixed Fw to allow Bonding with Citrix platforms
5. Fixed FW to allow upgrades of version 4.x to 4.x+ without requiring cold
boot
6. Fixed UEFI to not require 2nd reset
7. Fixed UEFI driver version reported
8. Fixed FW to prevent LOMS at 10G from negotiating 1G with Blade switches
9. Fixed FW for BE3 to report correct values of ethernet stat “In Range” error
10. Fixed FW to boot from secondary iSCSI when selected
11. Fixed FW iSCSI boot failure from secondary target when primary target given
is incorrect
12. Fixed UEFI NIC default settings retention
13. Fixed FW hang during boot with SLES 10 SP4
14. Fixed FW error while performing Shared Uplink Set port remove/add test
Known Issues
1. Firmware 4.2.401.6 is not compatible with Rhel 6.2×64 Inbox NIC driver
(version 4.1.255.11). In order to upgrade to Firmware 4.2.401.6, you must first
upgrade to Rhel 6.2×64 driver version 4.2.392.1 or later.
Symptom seen with 4.1.255.11 inbox driver: “TX hang in standard error as well as
/var/log/messages”
HP really needs to get their act together here. This is one of many reasons why Cisco UCS is so appealing. While certainly not bug free either, none of the “issues” with UCS firmware have ever impacted any my prod environments in any way. Can’t say the same for HP. QA really seems to be lacking here.
I posted on a similar link re the Emulex 4.2.401.2215 firmware but will also add the same message to this page, because it might be missed. So here goes the same question again:
Does anyone know what the new driver versions will be for ESXi 4.x and 5.x to be matched up to this new Emulex firmware 4.2.401.2215?
I am looking for the specific driver versions for the Ethernet (be2net) and FCoE (lpfc820) drivers.
Thanks in advance!
And here it goes.. new Driver for MBOX timeout’s…
http://h20565.www2.hp.com/portal/site/hpsc/template.PAGE/public/kb/docDisplay/?spf_p.tpst=kbDocDisplay&spf_p.prp_kbDocDisplay=wsrp-navigationalState%3DdocId%253Demr_na-c03844441-1%257CdocLocale%253D%257CcalledBy%253D&javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken
since 2 years only problem’s with these G7’s….
Hi Nuvagenic,
Try this shorter link:
http://h20564.www2.hp.com/portal/site/hpsc/public/kb/docDisplay/?docId=emr_na-c03844441
Note that these Emulex firmware versions are constantly rolling, due to bug fixes and enhancements. All vendors that use this CNA are subject to the same issues, not just HP with the G7 blade servers.