WoodITWork.com

HP has released an advisory saying that with versions of the Emulex Nic driver firmware pre 4.1.450.7 Device Control Channel (DCC) may not work which could cause your blades with NC55x Nics to drop off the network if the upstream network connection fails. This has been an issue which has plagued HP blade networking since the beginning and was a particular problem with the G6 Broadcom Nics so its disconcerting to see similar issues resurface.

http://h20566.www2.hp.com/portal/site/hpsc/template.PAGE/public/kb/docDisplay/?javax.portlet.endCacheTok=com.vignette.cachetoken&javax.portlet.prp_ba847bafb2a2d782fcbb0710b053ce01=wsrp-navigationalState%3DdocId%253Demr_na-c03600027-2%257CdocLocale%253D%257CcalledBy%253D&javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.tpst=ba847bafb2a2d782fcbb0710b053ce01&ac.admitted=1369727818792.876444892.199480143

4.1.450.7  is older firmware however and my post HP Virtual Connect Flex-10 & VMware ESX(i) pre-requisites recommends Emulex firmware version 4.2.401.605 so if you haven’t updated, now is the time.

HP blade servers have hard wired connections to their immediate upstream Virtual Connect switch via the chassis backplane. This means in normal operation the Nic on the blade will never see a down state  unless the physical Virtual Connect switch is powered off or removed. This would still be the case even if the upstream network connection failed if it wasn’t for DCC. DCC is a firmware component where Virtual Connect sees all uplinks for an Ethernet Network are down and then sends a command down to the individual Flex Nics to tell them to mark the port as down so the blade can fail over traffic to another nic in a port group or team. DCC  is what SmartLink does and requires coordination between Virtual Connect firmware and the blade Nic firmware which in the past has caused a lot of issues.

Something I noticed in the advisory however was you can actually now see whether DCC/SmartLink is working for each individual network connection. Navigate to the Interconnect Bays in the VCM GUI, then navigate to the server ports and view the DCC state:

{Source: HP}

This looks like a recent addition to Virtual Connect firmware and will certainly help troubleshooting network connection issues but hopefully the DCC disconnect debacle won’t be repeated.

VMware and HP have both released advisories which say you shouldn’t use HP Virtual Connect in tunnel mode if you are using vCloud Director Network Isolation (vCDNI) which is MAC-in-MAC encapsulation.

There are two network modes available with Virtual Connect, Tunnel Mode and Mapped Mode.

When using Tunnel Mode, Virtual Connect passes all tagged and untagged packets through the Virtual Connect switch down to select blades where the VLANs are split into port groups. The uplinks are therefore considered dedicated uplinks as control over which VLANs are trunked is done at the upstream switch and so you can’t have a different set of VLANs going to Blade 1 and Blade 2 while still utilising the same uplinks. You could obviously have separate sets of uplinks for Blade 1 and Blade 2 to achieve this. The advantage of tunneling mode is only having to specify your VLANs once at the upstream switch and being able to pass all VLANs down the same trunk to multiple blades and only having to manage VLANs at the upstream switch and port groups on the ESXi host or within the vSphere Distributed Switch.

In Mapped Mode  mode, the Virtual Connect switch examines all the VLANs and by defining Ethernet Networks for each VLAN on the Virtual Connect switch you can selectively pass down all or some of the VLANs down to the blades. The uplinks are considered shared as you can trunk all VLANs you will need for any blade and for example have some VLANs going to Blade 1 and other VLANs going to Blade 2 but sharing the same uplinks. In order to do this you  have to create separate Ethernet Networks for every VLAN (possibly two for redundancy) and manage VLANs at both the upstream and Virtual Connect switches as well as port groups on the ESXi host or within the vSphere Distributed Switch.

Read more…

HP has updated its ESXi 5.x customised images as part of its Service Pack for Proliant (SPP) Feb 2013 update.

HP’s customised images are fully integrated sets of specific drivers and software that are tested to work together. You can see the list of Driver Versions in HP supplied VMware ESX/ESXi images.

I have done an extensive update of my HP Virtual Connect Flex-10 & VMware ESX(i) pre-requisites post which includes these new customised images.

HP Custom Image for VMware ESXi 5.1 February 2013: VMware-ESXi-5.1.0-799733-HP-5.34.23.iso

The new and updated features for the HP vSphere 5.1 Customized Image for February 2013 include:

• Provider Features:
  • Report installed licenses for HP Dynamic Smart Array Controller.
  • Report New memory properties.
  • Support for IP Address encoding in SNMP traps.
  • Support SMX MemoryModuleOnBoard association.
  • HP Dynamic Smart Array Controller split cache support.
  • Report New RAID levels for storage volume fault tolerance.
  • HP Smart Cache support.
  • Update reporting of Smart Array Cache Status to align with firmware and iLO.
• AMS features:
  • Report running SW processes to HP Insight Remote Support.
  • Report vSphere 5.1 SNMP agent management IP and enable VMware vSphere 5.1 SNMP agent to report iLO4 management IP.
  • IML logging for NIC, and SAS traps.
  • Limit AMS log file size and support log redirection as defined by the ESXi host parameter: ScratchConfig.ConfiguredScratchLocation
• SR-IOV Support:
  • Updated Intel 10Gb network driver to enable SR-IOV for the HP 560FLB, 560M, 560SFP+, and 560FLR-SFP+.

HP Custom Image for VMware ESXi 5.0.0 Update 2 February 2013: VMware-ESXi-5.0.0-Update2-914586-HP-5.27.20.iso

The new and updated features for the HP vSphere 5.0 U2 Customized Image for February 2013 include:

• Provider Features:
  • Report installed licenses for HP Dynamic Smart Array Controller.
  • Report New memory properties.
  • Support for IP Address encoding in SNMP traps.
  • Support SMX MemoryModuleOnBoard association.
  • HP Dynamic Smart Array Controller split cache support.
  • Report New RAID levels for storage volume fault tolerance.
  • HP Smart Cache support.
  • Update reporting of Smart Array Cache Status to align with firmware and iLO.
• AMS features:
  • Report running SW processes to HP Insight Remote Support.
  • IML logging for NIC, and SAS traps.
  • Limit AMS log file size and support log redirection as defined by the ESXi host parameter: ScratchConfig.ConfiguredScratchLocation

Emulex has released a new set of drivers and firmware for use on Gen8 and G7 HP Servers including blades with both VMware ESXi 5.0 and 5.1.

Driver:
Emulex be2net nic driver version 4.2.327.0 which is included in the HP Custom Image for VMware ESXi 5.1 February 2013: VMware-ESXi-5.1.0-799733-HP-5.34.23.iso and HP Custom Image for VMware ESXi 5.0.0 Update 2 February 2013: VMware-ESXi-5.0.0-Update2-914586-HP-5.27.20.iso or can be downloaded separately from VMware.

Emulex has also released a newer driver version 4.4.231.0 which is available from VMware but this isn’t supported as yet by HP.

Firmware:
Emulex firmware version 4.2.401.605 which can be updated using the Emulex OneConnect-Flash .ISO.

There was a 4.2.401.6 released but this has been superseded by 4.2.401.605 which fixes a number of issues.

I have done an extensive update of my HP Virtual Connect Flex-10 & VMware ESX(i) pre-requisites post which includes these new Emulex drivers and firmware.

I have had the pleasure today of presenting at the second  UK VMware User Group. at the National Motorcycle Museum in Solihull, near Birmingham.

My presentation was titled vSphere Networking and Converged IO with Blade Servers.

I discussed how converged networking incorporating 10GE and FCoE is become more important as we try to utilise bandwidth more efficiently which is especially important with blade servers which have limited IO expansion. I explained the architecture of Cisco UCS and HP Virtual Connect as examples and how they can work with vSphere.

I went through some of the IO convergence options and also how to still protect your traffic by using either physical or logical separation or software.

No need to repeat everything else in this blog post, you may as well have a look at the presentation and best of all plan to attend a VMUG near you!

HP Blade chassis are integral to many IT infrastructure deployments. Within each chassis there are multiple components: for example Onboard Administrators, network switches running maybe Virtual Connect and the blade servers themselves which contain SSD/HDD drives, Nic or HBA cards, iLOs and motherboards.

All of these components require firmware to run and many of them are from different manufactures such as Nic cards which can be made by Broadcom, Intel or Emulex. Managing this firmware can be a frustrating and time consuming process which adds another management layer separate from the Operating System, applications and even HP drivers installed on the actual blade servers.

HP has previously had a very disparate way of managing it’s firmware and drivers. Different firmware and drivers had to be updated with different tools and was often very little or no coordination between the different updates causing any number of issues.

HP seems to have responded to customer frustration and is now coordinating their firmware and driver update procedures.

Read more…

HP has released a new firmware version 4.1.450.7 of its Emulex OneConnect 10Gb Ethernet Controller which is used in HP G7 and G8 Blades.

http://bizsupport1.austin.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=12169&prodSeriesId=4118472&prodNameId=4118473&swEnvOID=54&swLang=8&mode=2&taskId=135&swItem=co-108044-1

This is a standalone image so not part of the HP Service Pack for ProLiant 2012.08.0 release which is unfortunate.

The new firmware release notes state the following:

Added support for the following operating systems:

• RHEL 6.3

Resolved Issues from Firmware Version 4.1.402.20:

• BladeEngine Firmware
• SR IOV: Unable to ping IPV6 from DUT VF to Peer VF
• Running ethtool with BE3 takes longer than expected after cable pull
• FAT_CFG_SNAPSHOT command timed out
• Out of Order CQEs
• Data integrity Issue noted during short duration cable pulls
• W2K8 graceful shutdown failed under B120i with NC552
• BL460c Gen8 with 554FLB Unable to PXE boot
• 1Gb LOM ports in Windows do not come back up when switch is power cycled
• RSCN is not generated by switch when a driver restarts 10. 7 second delay in _CARD_ATTRIBUTES command
• BE3 read sfp ioctl returns bad data on twinax cables that BE2 returns good data with
• 554FLR FCoE Path Multipath Recovery Fail SLES11 sp1 x64
• Out of order Abort completion and IO completion CQE
• Received CQE_EXT_INVALID_RPI in abort completion of ELS command
• Correction to OCSD to prevent DMA conflicts with LPFC driver

PXE BIOS

• PXE issue on Gen 8 blades hang at gPXE prompt when set for one time PXE boot.

UEFI iSCSI

• EFI: CHAP name only accepts 6 to 10 characters
• EFI: iSCSI name is case sensitive

I’ve updated my post: HP Virtual Connect Flex-10 & VMware ESX(i) pre-requisites

Virtual Connect

HP has released its latest Virtual Connect Firmware which is used in HP c-Class Chassis with Flex-10, FlexFabric, VC Enet and FC Module interconnect modules.

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&swLang=8&mode=2&taskId=135&swItem=MTX-d3c2c57d850f4f4bb9af74c2d6

This release adds support for some new hardware:

• HP ProLiant BL660c Gen8 Server series
• HP ProLiant WS460c Gen8 Workstation series
• HP Virtual Connect Flex-10/10D Module
• HP 7m C-series Active Copper SFP+ cables (QK701A)
• HP 10m C-series Active Copper SFP+ cables (QK702A)
• Cisco 7m copper active Twinax cables (SFP-H10GB-ACU7M)
• Cisco 10m copper active Twinax cables (SFP-H10GB-ACU10M

You can now rollback from a previous firmware upgrade without having to delete the VC domain which is good news.

There are also number of management enhancements such as Advanced telemetry and statistics for Link Aggregation Groups and FlexNICs and Display of factory default MACs and WWNs in server profiles.

Read more…

HP has released a new firmware version 4.1.402.20 of its Emulex OneConnect 10Gb Ethernet Controller which is used in HP G7 Blades.

The June HP Service Pack for ProLiant 2012.06.0(B) had a 4.1.402.8 firmware version so it looks like it only had a lifespan of a month and has now been superseded by 4.1.402.20.

Unfortunately it doesn’t look like the HP VMware Recipe for June has been updated http://vibsdepot.hp.com/hpq/recipes/June2012VMwareRecipe1.0.pdf

The installation is as before and requires you to boot off an .ISO and update manually. (I have a dream that one day…)

Before June the firmware version recommended was 4.0.493. I can’t tell what the enhancements are between 4.1.402.8 and 4.1.402.20 as the release notes for 4.1.402.20 only list the resolved issues from 4.0.493.

BladeEngine Firmware

1. Increased RSS queues for Rhel drivers.
2. Fixed W2K8 graceful shutdown on NC552 adapter.
3. iSCSI not working when priority fields are used.
4. 554FLR-SFP adapter, FCoE Path Multipath Recovery Failure, SLES11 SP1 x64
5. FCoE port sometimes doesn’t login after VC module failover, w2k3
6. Fix support for TEMP_DEAD and ALOM_FAN_ON_AUX on Gen 8 servers
7. Correction to OCSD to prevent DMA conflicts with LPFC driver

PXE BIOS

1. NC553i adapter using wrong IP for gateway during Tftp
2. BE2/BE3 based adapters, PXE Hang
3. BL465cG7 PXE boot stops and is not coming up in RDP

iSCSI BIOS

1. SCSI SAN boot fails when DHCP Vendor Id is used for BL420cG8
2. Smart Array shows EBDA Error when ELX iSCSI Boot LUN configured

UEFI NIC

1. Dynamic link update is not working with NIC UEFI driver

Technical Tips

1. When updating HP CN1000e and NC550SFP products, use either the OneConnect-flash.iso or the Elxflash Offline tools. Either method will update the ILO image for the Network Communications Services Interface (NCSI) side band support. This procedure requires a power cycle after flashing. This tip does not apply to network cards that are version 4.0.360.15 or later.
2. Before you update your host drivers, update your firmware using the available online tools.

I’ve updated my post: HP Virtual Connect Flex-10 & VMware ESX(i) pre-requisites

HP has released a new version of its Virtual Connect Firmware 3.60 which is used in HP c-Class Chassis with Flex-10, FlexFabric, VC Enet and FC Module interconnect modules.
http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=3709945&prodSeriesId=3794423&prodNameId=3794431&swEnvOID=2079&swLang=8&mode=2&taskId=135&swItem=MTX-f8ab0a5ea5c8462390b90a1986

This release adds support for the following server models, HBAs and Adapters:

• HP ProLiant BL420c Gen8 Server Series
• HP ProLiant BL465c Gen8 Server Blade
• HP LPe1205A 8Gb FC HBA for BladeSystem c-Class
• HP Ethernet 1Gb 2-port 361FLB Adapter
• HP Ethernet 1Gb 4-port 366M Adapter

This update has fixed quite a number of bugs, 14 are listed in the release notes but there are 7 new issues reported so make sure you read the notes carefully.

There are also quite a few general issues HP is tracking even with this new firmware, some of which have workarounds and others with promised fixes in future versions.

Also make sure you check through the pre-requisites to ensure you follow some of the advisories or you may lose configuration data and ensure you use the Virtual Connect Support Utility 1.6.0