$NTPServers = "10.1.1.1","10.1.1.2"
Get-Cluster "MyCluster*" | Get-VMHost | Sort Name | %{
$_ | Remove-VMHostNtpServer -NtpServer ($_ | Get-VMHostNtpServer) -Confirm:$false
$_ | Add-VMHostNtpServer -NtpServer $NTPServers
$_ | Get-VMHostService | Where-Object {$_.key -eq "ntpd"} | Restart-VMHostService -Confirm:$false
}
It looks like the continuing HP Blade / VMWare compatibility issues with the Broadcom Nic drivers are ongoing as ESX 4.1 now can’t use HPs Smart Link and DCC.
In my previous post I listed the Flex-10 ESX pre-requisites which was for ESX 4.0 and works with Smart Link and DCC.
http://www.wooditwork.com/2010/08/09/flex-10-esx-pre-requisites/
Let me explain how one of the components of Flex-10 was designed for network failover.
Flex-10 switches have a great sounding feature called Smart Link. When the uplink ports from your Flex-10 switches in your chassis show a down link Smart Link knows that the upstream network is unavailable and in turn marks the individual blade nic down. Your ESX hosts would see a down nic with a red cross and fail over to the other nic(s) in the ESX team.
Smart Link is enabled within the Virtual Connect Manager | Ethernet Networks
When this first came out HP didn’t really explain that for this to actually work as advertised every uplink for every Flex-Nic connected to one of the 10Gb Broadcom Nics would have to be down for the blade nic to see it as down. It couldn’t work just on a single Flex-Nic.
If you had any internal networks defined mapped to Flex-Nics through your server profiles these would never fail as they were only internal to your Virtual Connect domain and so your Flex-Nics could never show a down status.
Using Beacon Probing with ESX was the solution, in effect not using Smart Link at all. You can leave Smart Link enabled, it just won’t be able to use it.
Beacon probing sends out network packet probes over both nics to upstream switches outside the Flex-10 Virtual Connect network so can tell when an uplink is down.
Here’s VMWare’s take on Beacon Probing: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1005577 although they do talk about having at least 3 Nics in a team which I feel is not required, 2 will do and works.
Beacon Probing is configured on the vSwitch under Nic Teaming | Network Failover Detection.
In order to work on an individual Flex-Nic it would need a driver component update called Device Control Channel (DCC) which would manage the up/down state of an individual Flex-Nic within a 10Gb nic.
This was all brought to life when the Broadcom 1.48 driver was released which enabled DCC which was the version I recommended be installed post your ESX 4.0 install. Smart Link would now work down to a Flex-Nic level.
You needed to have compatible HP Virtual Connect firmware (minimum 2.3) and each blade nic firmware had to be at least 5.0.11. Best though to go for the versions in my original post.
I think its probably best to keep Beacon Probing configured even after upgrading the nic driver. If Smart Link doesn’t t work as expected Beacon Probing will still kick in adding another level of failover.
ESX 4.1 has been released and now has a new Broadcom driver version 1.54.
HP has released an advisory saying this driver version now doesn’t work with Smart Link and DCC…great..one step forwards, 2 steps back!
http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&objectID=c02476622&jumpid=reg_R1002_USEN
The resolution they offer is a workaround to use Beacon Probing until a newer driver can be released.
So if you are intending to upgrade to or use ESX 4.1 you need to have Beacon Probing enabled otherwise your Flex-Nics won’t fail over and just have to wait until the driver is updated again to get Smart Link to work.
VM performance troubleshooting is something every VM Admin needs to do.
Often these can be related to agents running within a VM and often these are set to run at a particular local time.
VDI deployments often have people from different time zones working different shifts connecting to VMs in a central site with the idea that a particular agent task will always run out of hours from when people work. This would obviously not happen if the guest local time zone was not set correctly.
PowerCLI to the rescue and a one-liner that connects to a particular set of VMs and uses WMI to find out what time zone is set.
Get-VM -Location (Get-Folder "Workstations-Call Center") | Where { $_.PowerState -eq "PoweredOn" } | select Name, @{ Name="Time Zone"; Expression={(Get-WmiObject -Class win32_TimeZone -Comp $_.Guest.HostName).Caption}}
Hugo Peeters has a great function for using WMI to get Logged on Users:
http://www.peetersonline.nl/index.php/powershell/oneliner-get-logged-on-users-with-powershell/
This can be easily used in PowerCLI. This can be very useful when you need to do some VM maintenance in a VDI environment and need to find out who is currently logged on and know which VMs are not in use and can be worked on.
$VMs = Get-Cluster "LON_PROD1" | Get-VM | Where { $_.PowerState -eq "PoweredOn" }
ForEach ($VM in $VMs) {
Write-Host "Logged On Users for: " $VM.Guest.HostName
Get-WmiObject Win32_LoggedOnUser -ComputerName $VM.Guest.HostName | Select Antecedent -Unique | %{"{0}\{1}" -f $_.Antecedent.ToString().Split(’"‘)[1], $_.Antecedent.ToString().Split(’"‘)[3]}
}
VMware’s Infrastructure Operations (VIOPS) website is an unfortunately underknown yet fantastic resource to find proven practice documents, vetted installation documents and plenty of other tips and tricks for various VMware products written by gurus from not only VMware but the community.
Definitaly worth having a look…
http://communities.vmware.com/community/viops
There are many occasions when you may need to produce a report showing some aspect of your VM environment.
Many companies have various types of inventory databases which grab information from various sources to provide tracking and reporting of the various IT assets.
An example of a database could be one which pulls information from the software inventory system, patch management system, anti-virus console showing latest file update, AD etc. which could show a dashboard view of the workstations and highlight where they are not in compliance to company standards.
As a virtual workstation environment grows there is plently of information from vCenter that would be very useful to gather.
Grouping VMs by VDI groups in vCenter folders so you know how many VMs x department has would be useful to have. For the dashboard view it may be usedful having the VM Tools status. It’s could also be useful knowing which VMs are in which cluster and datastore, what virtual CPU and RAM they have and how big their disks are which can be used for capacity planning, performance analysis and reporting on what is deployed where.
Here are some handy script lines for working out what CDs you have connected which can cause Vmotion to fail.
You don’t always want to disconnect all CDs. If you have .ISOs on shared storage Vmotion will be fine as long as the hosts can both see the .ISO files.
Get the names of VMs with connected CD drives:
get-vm | where { $_ | get-cddrive | where { $_.ConnectionState.Connected -eq "true" } } | select Name
Get the names of VMs with connected .ISOs
get-vm | where { $_ | get-cddrive | where { $_.ConnectionState.Connected -eq "true" -and $_.ISOPath -like "*.ISO*"} } | select Name, @{Name=".ISO Path";Expression={(Get-CDDrive $_).isopath }}
Disconnect all VMs where the CD Drive is connected and it is not an .ISO
$VMs = Get-VM
$CDConnected = Get-CDDrive $VMs | where { $_.ConnectionState.Connected -eq "true"} -and $_.ISOPath -notlike "*.ISO*"}
If ($CDConnected -ne $null) {Set-CDDrive -connected 0 -StartConnected 0 $CDConnected -Confirm:$false }
If you work on multiple systems and need to use powershell to manage them but they all have their own separate installation it can drive you mad having so many different console shortcuts to launch.
I have 3 at the moment, VMware vSphere PowerCLI, Citrix XenServer PowerShell SnapIn and NetApp’s Data ONTAP PowerShell Toolkit.
Each one installs slightly differently and then runs separately which is a pain.
VMware vSphere PowerCLI installs a PSSnapin and creates this shortcut:
C:\WINDOWS\system32\WindowsPowerShell\v1.0\powershell.exe -psc “C:\Program Files\VMware\Infrastructure\vSphere PowerCLI\vim.psc1″ -noe -c “. \”C:\Program Files\VMware\Infrastructure\vSphere PowerCLI\Scripts\Initialize-VIToolkitEnvironment.ps1\”"
Citrix XenServer PowerShell SnapIn installs a PSSnapin and creates this shortcut:
"C:\Program Files\Citrix\XenServerPSSnapIn\XenServerPSSnapIn.bat"
which runs
@C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -PSConsoleFile XenServerPSSnapIn.psc1 -Noexit -Nologo
NetApp’s Data ONTAP PowerShell Toolkit requires you to copy files to the Module Path and then use Import-Module to load the snap-in.
So, Profiles to the rescue……powershell runs a startup script called your profile which is just a text file really.
What I really would like is to have a single profile that launches my session on some computers and starts all 3 snap-ins if they are installed.
That’s not a big ask, is it?
Well, lets start with setting up the profile on this particular computer to launch all 3 snap-ins but be clever and allow it to also be run from anywhere.
Updated: 02 May 2012 to include updated G7 Emulex driver for ESX(i) 4.x
Many companies are currently going through major infrastructure changes as the reach of virtualisation into IT increases.
Blade server use is growing as people start to think more seriously about large server farms and building the infrastructure for a “cloud”, be it internal or external. HP has been doing a lot of work selling its blades (BL460C and BL490C) in C7000 chassis with Flex-10 switches.
These infrastructure changes can be pretty complicated to implement as you need to plan for a whole new server environmnet, not just adding another server into a rack somewhere.
In a converged world you also need to get server people, power people, VM people, storage people and networking people to work together which is a challenge at the best of times.
HP does however have some problem delivering the technical message to get this all done. Unfortunately there isn’t enough clear information from HP about what needs to be done. Their reference architectures and Flex-10 cookbook are incomplete or out of date and in some cases contradictory and misleading.
It doesn’t help that HP’s website must rate as one of the worst to find anything and it’s an IT company…quite a big one as well and it lets itself down when its hardware in my opinion is fantastic.
So, here’s a pre-requisite list I’ve been building up to hopefully help you get you on your way.
HP BladeSystem c-Class Onboard Administrator (OA) Firmware
Update all your chassis OA firmware to 3.32 unless you are using Gen 8 blades.
http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=329290&prodSeriesId=3188465&prodNameId=3188475&swEnvOID=4040&swLang=8&taskId=135&swItem=MTX-9f3e4cd27ab041d59b8c196d99&mode=3
If you are using Gen 8 blades you will have to update your OA firmware to at least 3.50 as although the blades will be recognised there will be issues with Virtual Connect. A newer version 3.55 was also released on the same day which fixed some authentication and EBIPA issues which must have slipped through the testing net with 3.50 (great!) . I haven’t tested 3.50/3.55 but you don’t have much choice if you are using Gen 8 blades do you!
OA 3.55 can be downloaded from:
http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=329290&prodSeriesId=3188465&swItem=MTX-1ee4cda0b03447c6974e4ca184&prodNameId=3188475&swEnvOID=4040&swLang=8&taskId=135&mode=3
HP BladeSystem c-Class Virtual Connect Ethernet (VC-Enet), VC 4Gb FC, VC 8Gb 20-port FC, and VC 8Gb 24-port FC modules
Update your Virtual Connect Flex-10 switches to firmware 3.51 as this is a major release with many bug fixes and is a requirement if you are using Gen 8 blades.
Virtual Connect firmware 3.30 was released but seemed to have some stability issues so has been superseded by 3.51.
If you are still using firmware pre-3.17 you need to be aware of two major issues:
HP Flex-10 issue and DNS causing network outages.
If you are using pre-3.17 firmware, see my post to implement a DNS workaround for a serious issue that can cause the Flex-10 switches to lose connectivity with each other.
HP Flex-10 3.15 firmware bug and PXE kills Virtual Connect.
If you are using 3.15 firmware, see my post about ensuring all your unused blades are powered off to avoid a bug which causes the Flex-10 switches to restart after a memory leak caused by blades continually PXE booting.
Blade Server Firmware
HP has recently changed the way it does firmware and drivers and opted for an integrated approach which it is calling HP Service Pack for ProLiant (SPP) . The idea is that 3-5 times a year HP will release a whole pack of firmware and software which should all be tested together for compatibility.You will be able to use HP Smart Update Manager (HP SUM) to look at an entire environment from HP Blades through to the Chassis OA & Virtual Connect and be able to see all interdependencies and update firmware across the whole lot in the correct order. These will now be named by the date they were released such as 2012.02.0 and hopefully always be downloadable from http://h18004.www1.hp.com/products/servers/service_packs/en/index.html
These SPPs will be supported for 1 year which means theoretically HP won’t force you to always upgrade to the latest drivers and firmware unless they can show that the problem you are experiencing is due to a firmware / software problem. There’s a whole bunch of documentation here which you can use to read up on it.
I haven’t had a chance to see how successful this approach is with the new SPP. As you are aware updating firmware and drivers is quite a challenge and once you find a set that works you generally stick with it until there is a problem so for now I’m only able to recommend the previous version but will update my recommendation once I can test it all out and be sure it all works as advertised.
Smart Update Firmware DVD 9.30 (2011.05.05)
Update your blade server firmware to the new automatic deployment DVD version 9 or your Nics may not be visible. Version 8.x of the firmware DVD required you to manually update components which is a pain. Boot off the DVD and it does everything automatically. Unfortunately if you run it through an iLO connection it disconnects during the update so just run it again to ensure it runs through. The host will just loop through reboot after automatic installation after reboot so once it’s been through the process the second time you are done and can disconnect the iLO DVD Drive.
http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&swLang=8&mode=2&taskId=135&swItem=MTX-94560dcc34e845edbb13a26792
HP has also recently changed the naming convention of their firmware update DVD. The previous release, 9.20 (B) is now referred to by the release date using a US date format 2010.12.2.
Thermal Shut down issue
There is also an issue with the HP blade firmware pre 9.20 (B) that can cause a false temperature alert reading which can cause blades to shut down.
If this happens en-masse you’re in big trouble. Best to turn off the thermal shutdown until a new firmware is out which fixes it.
For each blade, launch the System Management Homepage (not the iLO) which is normally https://servername:2381 and log on.
Under Recovery Click Environment
Select Continue as the Thermal Degraded Action and click Set
G6 Blades and VMware ESX/ESXi 4.0
There is a long standing issue with the ESX Broadcom bnx2 driver.
There have been quite a few drivers released, some of which support DCC/SmartLink and some that don’t.
The latest driver which supports DCC/SmartLink for ESX 4.0 is 1.62.11.v40.1
This driver can be downloaded from here:
http://downloads.vmware.com/d/details/dt_esxi40_broadcom_bcm57xxx/ZHcqYnR0d3BiZGVqdA==
The good news is this driver can now be uploaded into Update Manager and you can deploy the new bnx2x driver along with other ESX patches.
G6 Blades and VMware ESX/ESXi 4.1 Update 1/2
HP and VMware have finally tested and certified a driver that works with DCC/SmartLink and is available through Update Manager as the shipped ESX 4.1 drivers caused PSODs.
The latest driver which supports DCC/SmartLink for ESX 4.1 is 1.62.15.v41.2
This driver can be downloaded from here:
http://downloads.vmware.com/d/details/dt_esxi40_broadcom_bcm_netxtreme/ZHcqYnQlZXBiZGhwZA==
If you are upgrading from the previously supported driver 1.60.50.v41.2 the heap memory fix is included so you don’t need to remove the workaround as was described in the original issue. http://kb.vmware.com/kb/1031805
The good news is this driver can now be uploaded into Update Manager and you can deploy the new bnx2x driver along with other ESX patches.
G7 Blades and VMware ESX(i) 4.x
You need the Emulex be2net driver version 4.0.306.0 which can be downloaded from here:
http://downloads.vmware.com/d/details/dt_esxi40_emulex_be2net_403060/ZHcqYnQqaGpiZEBlZA
G6 Blades and VMware ESXi 5
It’s good news for G6 blades which use a Broadcom bnx2x network card. The in-box driver version 1.61.15.v50.1-1 is supported with Virtual Connect so if you have at least NIC firmware version 5.2.7 and Virtual Connect firmware3.1.0 there’s no additional patching required.
G7 Blades and VMware ESXi 5
G7 blades use an Emulex be2net Nic rather than the Broadcom bnx2x Nic used in G6 blades.
The inbox ESXi5 Emulex be2net driver 4.0.88.0 doesn’t work if you used VLAN tagging anywhere in your network path and also has issues with PXE which will affect you if you wanted to use Auto Deploy.
HP released a new driver version 4.0.355.1 which they then pulled as there were still issues with this driver and PXE.
HP then re-released the same driver but with a new Nic firmware requirement version 4.0.360.7. Although the numbers look very similar, 4.0.355.1 is the Nic driver version and 4.0.360.7 is the Nic firmware version.
HP then released a newer 4.0.360.20 firmware which was then pulled and replaced with 4.0.360.15a. Yes, I know .15a surely shouldn’t be released after .20!
There was however a potential known issue with the 4.0.360.15a firmware update procedure which is listed in the installation instructions:
If you receive the following error during the upgrade “Warning: One or more CRC errors was detected during the flash operation. Prior to restarting your system it is important to retry the flash update. A failed flash update may leave your system in an unknown state”
Perform the following for the adapters listed:
This firmware has now been updated to 4.0.493.0 which can be downloaded from the following link:
The 4.0.355.1 driver version can be downloaded from VMware’s site.
http://downloads.vmware.com/d/details/dt_esxi50_emulex_be2net_403551/dHRAYnQqQHBiZHAlJQ
This can be uploaded to Update Manager to patch your hosts or if you are using Auto Deploy you can build the driver into a custom ESXi 5.0 depot using VMware Image Builder.
To manually update each host you can enable ESXi troubleshooting mode as well as SSH. Extract the .vib file and copy it to the local /tmp folder and use esxcli to install.
esxcli software vib install -v /tmp/net-be2net-4.0.355.1-1OEM.500.0.0.406165.x86_64.vib
My previous post talked about the design of BR and how to script a simple and scalable recovery process for getting as many VMs as you need from a primary site to a BR site when using NFS storage.
When using iSCSI storage it is slightly more complicated as you can’t use one of the benefits of NFS in presenting a read only datastore to ESX hosts in your BR site and having your VMs in the inventory and ready to just power on.
The manual steps would be:
Break the storage mirror.
Rescan the storage on the ESX hosts in the recovery site to pick up the now-visible LUN.
Browse the LUN for all .vmx files and add them into the inventory
Power on the VMs and answer the UUID question that they have moved.
Thanks LucD who has two great versions of a function for importing .VMX files, much better than the old community extension method.
http://communities.vmware.com/message/1317271#1317271
http://communities.vmware.com/message/1270822#1270822
The function is very powerful so it’s worth keeping as a link for using elsewhere.
Recent Comments