Home > VMware > Beware, Linked Mode can delete your vCenter permissions and licenses!

Beware, Linked Mode can delete your vCenter permissions and licenses!

September 16th, 2010

VMware released a new KB article yesterday with some more linked mode recovery steps.  Linked mode issues can be pretty painful to troubleshoot and the resolution can cause you to lose vCenter information….so read on and be prepared!

http://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&externalId=1024329

So, what is linked mode if you are not aware?

Linked mode is a great feature of vCenter that allows you go connect separate vCenter instances together and share information. When you log on to any of the linked vCenter servers with your vSphere client, you are able to see the inventory of all the vCenters which is really useful.
http://pubs.vmware.com/vsp40_e/wwhelp/wwhimpl/js/html/wwhelp.htm#href=admin/c_using_vcenter_server_in_linked_mode.html

You are also able to use the search box in the top right corner and search for VMs across all your vCenter servers which is very useful when your environment grows. Users don’t need to remember a list of different vCenter servers to connect to.

License information is also shared between linked vCenter servers so you can have a single license key applied in one place and use the license count for any server in any vCenter instance.

Roles are also shared between vCenter servers so you can use and manage the same role across your enterprise and set permissions based on those roles.

VMware uses Microsoft’s Active Directory Application Manager (ADAM) which is a cut down version of Active Directory which applications can use to replicate information in the same way as AD.

Here’s a walk through for the installation:
http://searchvmware.techtarget.com/tip/0,289483,sid179_gci1380856_mem1,00.html

A common business recovery process is to use storage mirroring to replicate VM disk file data for servers including vCenter from a primary to secondary site to have vCenter available in a recovery situation.

Linked mode doesn’t like the vCenter Server IP address being changed which can happen when you fail it over in this way. The fix is normally to use the vCenter Server Linked Mode Configuration application and remove the server from the linked mode group and then re-add it which reestablishes the linked mode group and fixes the IP address confusion and your vCenters are back to sharing nicely.

Sometimes removing the secondary vCenter server from the linked mode group works correctly but you cannot then re-add it.
Checking the linked mode log file C:\Users\%installationaccount%\AppData\Local\Temp\jointool-0.log you get the following error message: vCenter Server linked Mode. Error 28039. Setup cannot join vCenter Server to the linked mode group.

Unfortunately there is not much information from VMware on ADAM troubleshooting. What is available starts with pretty basic connectivity checking. Can you ping the boxes, can you telnet on specific ports, is the service account locked out, checking ODBC, etc.

As ADAM pulls information from the SQL database before managing the actual replication, there could be an underlying problem with the SQL database which VMware may recommend be fixed. If the rest of vCenter is running correctly and the only problem is linked mode replication, the issues are probably not to do with the SQL database. A troubleshooting step with vCenter has sometimes been to reinstall vCenter over the top to “fix it” which is a process which I generally disagree with. Reinstalling vCenter over the top to fix an issue when vCenter has normally been running fine for ages just seems to be using a nuclear bomb to crack a nut…

There are some prerequisites for ADAM which I’ve been checking up on. You need to have a service account or separate service accounts for each vCenter server that is a local admin and vCenter admin on all of the servers in the linked mode group. Log on as this/these service accounts to do your troubleshooting and fixing. If you are using Windows 2008 R2 you need to ensure you right click and run the tool as Administrator.

You can also view the information from ADAM using ADSI Edit, just as you can with AD.

  • Click Start > All Programs > Administrative Tools > ADSI Edit.
  • Right-click ADSI Edit and click Connect to.
  • Under Connection Point, select Select or type a Distinguished Name or Naming context and type:
    dc=virtualcenter,dc=vmware,dc=int
  • Under Computer, select Select or type a domain or server, enter the vCenter Server’s hostname you would like to check the ADAM information, and click OK.
  • Navigate to OU=Instances in the left pane and you can see which servers are part of the group.

There is a log file generated during for ADAM installation which may point to some issues.
C:\Users\%installationaccount%\AppData\Local\Temp\jointool-0.log

One of the errors when you cannot join a linked mode group is “The source server is currently rejecting replication requests.”

There is an ADAM troublsehooting batch file launcher which is used to fix ADAM issues. There is no help file associated with this so you only have VMware’s knowledge base information to go on.

C:\Program Files (x86)\VMware\Infrastructure\VirtualCenter Server\jointool.bat

Whenever you run jointool.bat you need to stop the vCenter and VMwareVCMSDS Services first. The tool doesn’t automatically do this.
First step is to run jointool.bat recover on the secondary server. This doesn’t output anything to the screen so you don’t know if it has “fixed” anything but check the log file on both servers which may indicate something. C:\Users\%installationaccount%\AppData\Local\Temp\jointool-0.log

Start the vCenter and VMwareVCMSDS Services manually again and check whether replication is now working. If not run jointool.bat on the primary server as well.

VMware’s KB released yesterday goes through the process of deleting the ADAM database and initialising it again to fix replication. It does warn you to back up all your licenses but doesn’t mention custom roles. As linked mode also replicates vCenter roles, if these are deleted, any permissions based on these roles are also removed from your inventory. As your vCenter grows so will your permission structure and losing these can be far more catastrophic than losing licenses. I think it is unbelievable VMware has written vCenter in a way that when it links up and doesn’t manage its conflicts correctly it can delete roles.

VMware documentation says that if there are duplicate named roles when joining VCs together it will rename the duplicate roles.
http://pubs.vmware.com/vsp40_e/wwhelp/wwhimpl/js/html/wwhelp.htm#href=admin/c_reconciling_role_conflicts.html#1_8_8_10_1

I think this may be a a little simplistic. If you previously had linked mode working and had a role replicated across vCenters then this is not the same as having two separate vCenter servers linking up for the first time and the tool checking conflicts. It is probably to do with UUIDs being the same when they were previously linked and linked mode deleting and recreating the roles.

WARNING: MAKE A BACKUP OF YOUR ALL YOUR VCENTER LICENSES, ROLES AND PERMISSIONS BEFORE YOU ATTEMPT TO RECOVER ADAM.

If jointool.bat recover doesn’t fix Linked Mode and you have to delete the ADAM database and start again here are the steps
BUT READ THE WARNING

  • Take a VMware snapshot or backup of your vCenter and associated SQL Servers before starting any work
  • Log onto both vCenter Servers with their local admin Service Accounts
  • Stop the VMwareVCMSDS Service
  • Stop the VMware VirtualCenter Management Webservices Service
  • Stop the VMware VirtualCenter Server Service
  • Delete the C:\Program Files (x86)\VMware\Infrastructure\VirtualCenter Server\VMwareVCMSDS folder which holds the ADAM database
  • run jointool.bat init –name –vimURL https://<vcserver>:443/sdk –webServiceURL https://<vcserver>:8443/vws –force
  • Repeat the process on the master linked mode server to delete its ADAM database
  • Start the VMware VirtualCenter Server services on both vCenter servers
  • Start the VMware VirtualCenter Management Webservices on both vCenter servers
  • Start the VMwareVCMSDS Service on both vCenter servers
  • You now have isolated vCenter servers and two blank ADAM databases.
  • Join the second server to the primary server using the vCenter Server Linked Mode Configuration application.

When you logon you would immediately receive a message about licenses if they have been deleted. Luckily you have a backup.
Reapply your licenses. You may need to reconnect some hosts if they are complaining about licenses.

Next you should check your roles and permissions and repermission anything that is missing.

Hopefully VMware will be releasing more linked mode and ADAM troubleshooting steps as currently information is limited and a huge amount of hassle can be caused without knowing what can be lost before you start.

Categories: VMware Tags: , ,
Comments are closed.