$NTPServers = "10.1.1.1","10.1.1.2"
Get-Cluster "MyCluster*" | Get-VMHost | Sort Name | %{
$_ | Remove-VMHostNtpServer -NtpServer ($_ | Get-VMHostNtpServer) -Confirm:$false
$_ | Add-VMHostNtpServer -NtpServer $NTPServers
$_ | Get-VMHostService | Where-Object {$_.key -eq "ntpd"} | Restart-VMHostService -Confirm:$false
}
It looks like the continuing HP Blade / VMWare compatibility issues with the Broadcom Nic drivers are ongoing as ESX 4.1 now can’t use HPs Smart Link and DCC.
In my previous post I listed the Flex-10 ESX pre-requisites which was for ESX 4.0 and works with Smart Link and DCC.
http://www.wooditwork.com/2010/08/09/flex-10-esx-pre-requisites/
Let me explain how one of the components of Flex-10 was designed for network failover.
Flex-10 switches have a great sounding feature called Smart Link. When the uplink ports from your Flex-10 switches in your chassis show a down link Smart Link knows that the upstream network is unavailable and in turn marks the individual blade nic down. Your ESX hosts would see a down nic with a red cross and fail over to the other nic(s) in the ESX team.
Smart Link is enabled within the Virtual Connect Manager | Ethernet Networks
When this first came out HP didn’t really explain that for this to actually work as advertised every uplink for every Flex-Nic connected to one of the 10Gb Broadcom Nics would have to be down for the blade nic to see it as down. It couldn’t work just on a single Flex-Nic.
If you had any internal networks defined mapped to Flex-Nics through your server profiles these would never fail as they were only internal to your Virtual Connect domain and so your Flex-Nics could never show a down status.
Using Beacon Probing with ESX was the solution, in effect not using Smart Link at all. You can leave Smart Link enabled, it just won’t be able to use it.
Beacon probing sends out network packet probes over both nics to upstream switches outside the Flex-10 Virtual Connect network so can tell when an uplink is down.
Here’s VMWare’s take on Beacon Probing: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1005577 although they do talk about having at least 3 Nics in a team which I feel is not required, 2 will do and works.
Beacon Probing is configured on the vSwitch under Nic Teaming | Network Failover Detection.
In order to work on an individual Flex-Nic it would need a driver component update called Device Control Channel (DCC) which would manage the up/down state of an individual Flex-Nic within a 10Gb nic.
This was all brought to life when the Broadcom 1.48 driver was released which enabled DCC which was the version I recommended be installed post your ESX 4.0 install. Smart Link would now work down to a Flex-Nic level.
You needed to have compatible HP Virtual Connect firmware (minimum 2.3) and each blade nic firmware had to be at least 5.0.11. Best though to go for the versions in my original post.
I think its probably best to keep Beacon Probing configured even after upgrading the nic driver. If Smart Link doesn’t t work as expected Beacon Probing will still kick in adding another level of failover.
ESX 4.1 has been released and now has a new Broadcom driver version 1.54.
HP has released an advisory saying this driver version now doesn’t work with Smart Link and DCC…great..one step forwards, 2 steps back!
http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&objectID=c02476622&jumpid=reg_R1002_USEN
The resolution they offer is a workaround to use Beacon Probing until a newer driver can be released.
So if you are intending to upgrade to or use ESX 4.1 you need to have Beacon Probing enabled otherwise your Flex-Nics won’t fail over and just have to wait until the driver is updated again to get Smart Link to work.
VM performance troubleshooting is something every VM Admin needs to do.
Often these can be related to agents running within a VM and often these are set to run at a particular local time.
VDI deployments often have people from different time zones working different shifts connecting to VMs in a central site with the idea that a particular agent task will always run out of hours from when people work. This would obviously not happen if the guest local time zone was not set correctly.
PowerCLI to the rescue and a one-liner that connects to a particular set of VMs and uses WMI to find out what time zone is set.
Get-VM -Location (Get-Folder "Workstations-Call Center") | Where { $_.PowerState -eq "PoweredOn" } | select Name, @{ Name="Time Zone"; Expression={(Get-WmiObject -Class win32_TimeZone -Comp $_.Guest.HostName).Caption}}
Hugo Peeters has a great function for using WMI to get Logged on Users:
http://www.peetersonline.nl/index.php/powershell/oneliner-get-logged-on-users-with-powershell/
This can be easily used in PowerCLI. This can be very useful when you need to do some VM maintenance in a VDI environment and need to find out who is currently logged on and know which VMs are not in use and can be worked on.
$VMs = Get-Cluster "LON_PROD1" | Get-VM | Where { $_.PowerState -eq "PoweredOn" }
ForEach ($VM in $VMs) {
Write-Host "Logged On Users for: " $VM.Guest.HostName
Get-WmiObject Win32_LoggedOnUser -ComputerName $VM.Guest.HostName | Select Antecedent -Unique | %{"{0}\{1}" -f $_.Antecedent.ToString().Split(’"‘)[1], $_.Antecedent.ToString().Split(’"‘)[3]}
}
VMware’s Infrastructure Operations (VIOPS) website is an unfortunately underknown yet fantastic resource to find proven practice documents, vetted installation documents and plenty of other tips and tricks for various VMware products written by gurus from not only VMware but the community.
Definitaly worth having a look…
http://communities.vmware.com/community/viops
There are many occasions when you may need to produce a report showing some aspect of your VM environment.
Many companies have various types of inventory databases which grab information from various sources to provide tracking and reporting of the various IT assets.
An example of a database could be one which pulls information from the software inventory system, patch management system, anti-virus console showing latest file update, AD etc. which could show a dashboard view of the workstations and highlight where they are not in compliance to company standards.
As a virtual workstation environment grows there is plently of information from vCenter that would be very useful to gather.
Grouping VMs by VDI groups in vCenter folders so you know how many VMs x department has would be useful to have. For the dashboard view it may be usedful having the VM Tools status. It’s could also be useful knowing which VMs are in which cluster and datastore, what virtual CPU and RAM they have and how big their disks are which can be used for capacity planning, performance analysis and reporting on what is deployed where.
Here are some handy script lines for working out what CDs you have connected which can cause Vmotion to fail.
You don’t always want to disconnect all CDs. If you have .ISOs on shared storage Vmotion will be fine as long as the hosts can both see the .ISO files.
Get the names of VMs with connected CD drives:
get-vm | where { $_ | get-cddrive | where { $_.ConnectionState.Connected -eq "true" } } | select Name
Get the names of VMs with connected .ISOs
get-vm | where { $_ | get-cddrive | where { $_.ConnectionState.Connected -eq "true" -and $_.ISOPath -like "*.ISO*"} } | select Name, @{Name=".ISO Path";Expression={(Get-CDDrive $_).isopath }}
Disconnect all VMs where the CD Drive is connected and it is not an .ISO
$VMs = Get-VM
$CDConnected = Get-CDDrive $VMs | where { ($_.ConnectionState.Connected -eq "true") -and ($_.ISOPath -notlike "*.ISO*")}
If ($CDConnected -ne $null) {Set-CDDrive -connected 0 -StartConnected 0 $CDConnected -Confirm:$false }
or another slightly quicker and dirtier way thanks to commenter José de Alonso
Get-VM | Get-CDDrive | Where {$_.ISOPath -ne $null} | Set-CDDrive -NoMedia -Confirm:$false
If you work on multiple systems and need to use powershell to manage them but they all have their own separate installation it can drive you mad having so many different console shortcuts to launch.
I have 3 at the moment, VMware vSphere PowerCLI, Citrix XenServer PowerShell SnapIn and NetApp’s Data ONTAP PowerShell Toolkit.
Each one installs slightly differently and then runs separately which is a pain.
VMware vSphere PowerCLI installs a PSSnapin and creates this shortcut:
C:\WINDOWS\system32\WindowsPowerShell\v1.0\powershell.exe -psc “C:\Program Files\VMware\Infrastructure\vSphere PowerCLI\vim.psc1” -noe -c “. \”C:\Program Files\VMware\Infrastructure\vSphere PowerCLI\Scripts\Initialize-VIToolkitEnvironment.ps1\””
Citrix XenServer PowerShell SnapIn installs a PSSnapin and creates this shortcut:
"C:\Program Files\Citrix\XenServerPSSnapIn\XenServerPSSnapIn.bat"
which runs
@C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -PSConsoleFile XenServerPSSnapIn.psc1 -Noexit -Nologo
NetApp’s Data ONTAP PowerShell Toolkit requires you to copy files to the Module Path and then use Import-Module to load the snap-in.
So, Profiles to the rescue……powershell runs a startup script called your profile which is just a text file really.
What I really would like is to have a single profile that launches my session on some computers and starts all 3 snap-ins if they are installed.
That’s not a big ask, is it?
Well, lets start with setting up the profile on this particular computer to launch all 3 snap-ins but be clever and allow it to also be run from anywhere.
Updated: 28 October 2013 to include new HP September bundles and release of ESXi 5.5.
Many companies are currently going through major infrastructure changes as the reach of virtualisation into IT increases and they start moving towards a cloud model.
Blade server use is growing as people start to think more seriously about large server farms and building the infrastructure for a “cloud”, be it internal(private) or external (public). HP has been doing a lot of work selling its blades (BL460C and BL490C) in C7000 chassis with Flex-10 switches.
These infrastructure changes can be pretty complicated to implement as you need to plan for a whole new server hosting environment, not just adding another server into a rack somewhere.
In a converged world you also need to get server people, power people, VM people, storage people and networking people to work together which is a challenge at the best of times in traditional siloed enterprise IT departments.
HP has however had some problems over the years delivering the technical message to get this all done. Unfortunately there often wasn’t enough clear information from HP about what needs to be done. Their reference architectures and Flex-10 cookbook were incomplete or out of date and in some cases contradictory and misleading. This has improved somewhat over the years but it doesn’t help that HP’s website must rate as one of the worst to find anything and it’s an IT company…quite a big one as well and it lets itself down when its hardware in my opinion is fantastic.
So, here’s a pre-requisite list I’ve been building up and maintaining over the years to hopefully get you on your way.
HP BladeSystem c-Class Onboard Administrator (OA) Firmware
Update all your chassis OA firmware to 4.01. Although this is a v4 release the update is not nearly as dramatic as Virtual Connect 4. The Onboard Administrator upgrade from 3.71 to 4.01 just adds IPv6 support, new hardware models and a raft of bug fixes.
Update all your chassis OA firmware to 3.71.
Older version information:
There is a problem with Emulex firmware prior to version 4.1.450.7 that can result in SmartLink otherwise known as Device Control Channel (DCC) not working with a 10Gb physical link when you have full height blades. When you upgrade the OA, you may lose network connectivity. Update the NIC firmware and look at Customer Advisory c03600027 before upgrading the OA.
If you are using Gen 8 blades you will have to update your OA firmware to at least 3.50 which is the latest Recommended release as although the blades will be recognised there will be issues with Virtual Connect.
A newer version 3.55 was released the next day which fixed some authentication and EBIPA issues which must have slipped through the testing net with 3.50 (great!) . OA 3.50 can be downloaded from: http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=3709945&prodSeriesId=1844065&prodNameId=3692952&swEnvOID=4064&swLang=8&taskId=135&swItem=MTX-00b278b5e9194e7ab0ab362d55&mode=3
OA 3.55 can be downloaded from:
HP BladeSystem c-Class Virtual Connect Ethernet (VC-Enet), VC 4Gb FC, VC 8Gb 20-port FC, and VC 8Gb 24-port FC modules
Update your Virtual Connect Flex-10 switches to firmware 3.75 as this is a Recommended release with many bug fixes.
Virtual Connect 4.01 has been released which includes many more features and has also since been updated to 4.1 . It’s still relatively new so if you are going to use it, do your testing.
Upgrade Alert: If you have both single dense and double dense servers in your chassis and you are upgrading from Virtual Connect firmwares 3.10-3.60 up to Virtual Connect firmwares 3.70-3.75 you need to be aware of a serious issue where upgrading the firmware may cause your server profiles to be unassigned which will cause a network outage.
See this post: HP Virtual Connect Firmware upgrade may cause Server Profiles to be unassigned causing a network outage
VC 3.75 can be downloaded from: http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=3709945&prodSeriesId=3794423&prodNameId=3794431&swEnvOID=4091&swLang=8&mode=2&taskId=135&swItem=MTX-a94c23fca5ac4a21a3b88aa931
If you are using Gen 8 blades you need at least firmware version 3.51. VC 3.51 can be downloaded from: http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=3709945&prodSeriesId=3794423&prodNameId=3794431&swEnvOID=4091&swLang=8&taskId=135&swItem=MTX-bd78dea478694b249c2a637281&mode=3
If you are still using firmware pre-3.17 you need to be aware of two major issues:
HP Flex-10 issue and DNS causing network outages.
If you are using pre-3.17 firmware, see my post to implement a DNS workaround for a serious issue that can cause the Flex-10 switches to lose connectivity with each other.
HP Flex-10 3.15 firmware bug and PXE kills Virtual Connect.
If you are using 3.15 firmware, see my post about ensuring all your unused blades are powered off to avoid a bug which causes the Flex-10 switches to restart after a memory leak caused by blades continually PXE booting.
HP Service Pack for ProLiant
HP changed the way it does firmware and drivers and opted for an integrated approach which it is calling HP Service Pack for ProLiant (SPP) . The idea is that 3-5 times a year HP will release a whole pack of firmware and software which should all be tested together for compatibility.
You will be able to use HP Smart Update Manager (HP SUM) to look at an entire environment from HP Blades through to the Chassis OA & Virtual Connect and be able to see all interdependencies and update firmware across the whole lot in the correct order. These will now be named by the date they were released such as 2012.02.0 and hopefully always be downloadable from:
http://h18004.www1.hp.com/products/servers/service_packs/en/index.html
These SPPs will be supported for 1 year which means theoretically HP won’t force you to always upgrade to the latest drivers and firmware unless they can show that the problem you are experiencing is due to a firmware / software problem. There’s a whole bunch of documentation here which you can use to read up on it.
Unfortunately the SPP software isn’t supported for all operating systems but will be the way going forward. OS Support is for: Windows Server 2008 and later, RHEL 5 and later and SLES 10 and later.
If you are using ESXi however, you will not be actually installing any SPP software on the host but just using the firmware components so don’t get too bogged down by OS supportability.
The latest SPP is 2013.09 and can be downloaded from: http://h18004.www1.hp.com/products/servers/service_packs/en/index.html
From the 2013.02 release you can also have online updating for particular server models of System ROM, iLO, SmartArray Controller and Drive firmware with ESXi 5.x using what HP are calling Smart Components which can also use Update Manager giving you a much simpler firmware update process for the supported firmware packages.
You can see the full supported list here: http://vibsdepot.hp.com/hpq/SmartComponents/
Smart Update Firmware DVD
If you are not using ESXi or OS versions on the SPP support list you need to use the old Smart Update Firmware DVD.
The current updated version is 9.30 (2011.05.05)
Update your blade server firmware to the automatic deployment DVD version 9 or your Nics may not be visible.
Version 8.x of the firmware DVD required you to manually update components which is a pain. Boot off the DVD and it does everything automatically. Unfortunately if you run it through an iLO connection it disconnects during the update so just run it again to ensure it runs through. The host will just loop through reboot after automatic installation after reboot so once it’s been through the process the second time you are done and can disconnect the iLO DVD Drive.
You can download 9.30 (2011.05.05) from:
Gen 8 and G7 blades use an Emulex Nic. ESXi versions up to 5.1 have the driver called be2net. ESXi 5.5 has a new drive model and so the Emulex drivers are now called elxnet.
G6 blades use a Broadcom bnx2x Nic for all ESXi versions.
There are two separate but related components that you need to ensure are in sync. The Nic driver software version that runs in ESXi and the Nic firmware software version which is updated separately.
HP creates ESX(i) bundles which contain supported HP software versions for various ESX(i) versions. As of Feb 2013 you can also update some firmware online from ESXi but this doesn’t include Nic firmware.
Nic Driver software can be downloaded as a .VIB file and be uploaded to Update Manager to patch your hosts or if you are using Auto Deploy you can build the driver into a custom ESXi 5.x depot using VMware Image Builder.
To manually update each host you can enable ESXi troubleshooting mode as well as SSH. Extract the .vib file and copy it to the local /tmp folder and use esxcli to install.
Example:
esxcli software vib install -v /tmp/net-be2net-4.0.355.1-1OEM.500.0.0.406165.x86_64.vib
VMware ESXi 5.5
Gen8 / G7 Blades
You need to be using Emulex elxnet nic driver version 10.0.575.7 which is included in the HP Custom Image for VMware ESXi 5.5 September 2013 or can be downloaded from VMware.
You need to be using Emulex firmware version 4.6.247.5 which can be updated using the Emulex OneConnect-Flash .ISO.
There is a problem with Emulex firmware prior to version 4.1.450.7 that can result in SmartLink otherwise known as Device Control Channel (DCC) not working with a 10Gb physical link when you have full height blades. When you upgrade the OA, you may lose network connectivity. Update the NIC firmware and look at Customer Advisory c03600027 before upgrading the OA.
G6 Blades
You need to be using Broadcom bnx2x nic driver version 1.78.28.v55.2 which is included in the HP Custom Image for VMware ESXi 5.5 September 2013 or can be downloaded from VMware.
You need to be using Broadcom firmware version 6.2.26 which can be updated using the HP Service Pack for ProLiant version 2013.09.0(b)
VMware ESXi 5.1
Gen8 / G7 Blades
You need to be using Emulex be2net nic driver version 4.6.247.10 which is included in the HP Custom Image for VMware ESXi 5.1.0 Update 1 September 2013 or can be downloaded separately from VMware.
You need to be using Emulex firmware version 4.6.247.5 which can be updated using the Emulex OneConnect-Flash .ISO.
There is a problem with Emulex firmware prior to version 4.1.450.7 that can result in SmartLink otherwise known as Device Control Channel (DCC) not working with a 10Gb physical link when you have full height blades. When you upgrade the OA, you may lose network connectivity. Update the NIC firmware and look at Customer Advisory c03600027 before upgrading the OA.
G6 Blades
You need to be using Broadcom bnx2x nic driver version 1.74.17.v50.1 which is included in the HP Custom Image for VMware ESXi 5.1.0 Update 1 September 2013 or can be downloaded separately from VMware
You need to be using Broadcom firmware version 6.2.26 which can be updated using the HP Service Pack for ProLiant version 2013.09.0(b)
VMware ESXi 5.0
Gen8 / G7 Blades
The inbox ESXi5.0 Emulex be2net driver 4.0.88.0 doesn’t work if you used VLAN tagging anywhere in your network path and also has issues with PXE which will affect you if you wanted to use Auto Deploy.
You need to be using Emulex be2net nic driver version 4.6.247.07 which is included in the HP Custom Image for VMware ESXi 5.0.0 Update 3 October 2013 or can be downloaded separately from VMware.
You need to be using Emulex firmware version 4.6.247.5 which can be updated using the Emulex OneConnect-Flash .ISO.
There is a problem with Emulex firmware prior to version 4.1.450.7 that can result in SmartLink otherwise known as Device Control Channel (DCC) not working with a 10Gb physical link when you have full height blades. When you upgrade the OA, you may lose network connectivity. Update the NIC firmware and look at Customer Advisory c03600027 before upgrading the OA.
G6 Blades
You need to be using Broadcom bnx2x nic driver version 4.2.327.0 which is included in the HP Custom Image for VMware ESXi 5.0.0 Update 2 February 2013: VMware-ESXi-5.0.0-Update2-914586-HP-5.27.20.iso or can be downloaded separately from VMware
You need to be using Broadcom firmware version 6.2.26 which can be updated using the HP Service Pack for ProLiant version 2013.09.0(b)
If you are having any issues with previous drivers:
After the inbox driver, HP released a new driver version 4.0.355.1 which they then pulled as there were still issues with this driver and PXE.
HP then re-released the same driver but with a new Nic firmware requirement version 4.0.360.7. Although the numbers look very similar, 4.0.355.1 is the Nic driver version and 4.0.360.7 is the Nic firmware version.
HP then released a newer 4.0.360.20 firmware which was then pulled and replaced with 4.0.360.15a. Yes, I know .15a surely shouldn’t be released after .20!
There was however a potential known issue with the 4.0.360.15a firmware update procedure which is listed in the installation instructions:
If you receive the following error during the upgrade “Warning: One or more CRC errors was detected during the flash operation. Prior to restarting your system it is important to retry the flash update. A failed flash update may leave your system in an unknown state”
Perform the following for the adapters listed:
VMware ESX/ESX(i) 4.1 Update 1/2
Gen8 / G7 Blades
You need the Emulex be2net driver version 4.1.334.0 which can be downloaded from here: http://downloads.vmware.com/d/details/dt_esxi40_emulex_be2net_403060/ZHcqYnQqaGpiZEBlZA
You need the Emulex firmware version 4.1.402.20 which can be downloaded from here: http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=lamerica_nsc_carib&prodTypeId=329290&prodSeriesId=4324629&swItem=co-106538-1&prodNameId=4324630&swEnvOID=54&swLang=8&taskId=135&mode=4&idx=1
There is a problem with Emulex firmware prior to version 4.1.450.7 that can result in SmartLink otherwise known as Device Control Channel (DCC) not working with a 10Gb physical link when you have full height blades. When you upgrade the OA, you may lose network connectivity. Update the NIC firmware and look at Customer Advisory c03600027 before upgrading the OA.
G6 Blades
HP and VMware have finally tested and certified a driver that works with DCC/SmartLink and is available through Update Manager as the shipped ESX 4.1 drivers caused PSODs.
The latest driver which supports DCC/SmartLink for ESX 4.1 is 1.70.34.v41.1
This driver can be downloaded from VMware and deployed using Update Manager.
If you are upgrading from the previously supported driver 1.60.50.v41.2 the heap memory fix is included so you don’t need to remove the workaround as was described in the original issue. http://kb.vmware.com/kb/1031805
VMware ESX/ESX i 4.0
Gen8 / G7 Blades You need the Emulex be2net driver version 4.1.334.0 which can be downloaded from here: http://downloads.vmware.com/d/details/dt_esxi40_emulex_be2net_403060/ZHcqYnQqaGpiZEBlZA
You need the Emulex firmware version 4.1.402.20 which can be downloaded from here: http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=lamerica_nsc_carib&prodTypeId=329290&prodSeriesId=4324629&swItem=co-106538-1&prodNameId=4324630&swEnvOID=54&swLang=8&taskId=135&mode=4&idx=1
G6 Blades
There is a long standing issue with the ESX Broadcom bnx2 driver. There have been quite a few drivers released, some of which support DCC/SmartLink and some that don’t.
The latest driver which supports DCC/SmartLink for ESX 4.0 is 1.62.11.v40.1
This driver can be downloaded from VMware and deployed using Update Manager.
My previous post talked about the design of BR and how to script a simple and scalable recovery process for getting as many VMs as you need from a primary site to a BR site when using NFS storage.
When using iSCSI storage it is slightly more complicated as you can’t use one of the benefits of NFS in presenting a read only datastore to ESX hosts in your BR site and having your VMs in the inventory and ready to just power on.
The manual steps would be:
Break the storage mirror.
Rescan the storage on the ESX hosts in the recovery site to pick up the now-visible LUN.
Browse the LUN for all .vmx files and add them into the inventory
Power on the VMs and answer the UUID question that they have moved.
Thanks LucD who has two great versions of a function for importing .VMX files, much better than the old community extension method.
http://communities.vmware.com/message/1317271#1317271
http://communities.vmware.com/message/1270822#1270822
The function is very powerful so it’s worth keeping as a link for using elsewhere.
Recent Comments