VMware VirtualCenter Server service won’t Start November 24, 2009Posted by General Zod in Storytime, Tech, VMware.
Yesterday afternoon, I was minding my own business and working through a CBT, when I noticed that my personal VMware Infrastructure Client was suddenly disconnected. This is nothing new as (for some unknown reason I have yet to fathom) the VMware VirtualCenter Server service on my VCenter server will crash about once a month. It’s not too big of a deal since it usually starts right up again and all’s right with the world… except for yesterday.
Attempting to restart the service yielded nothing. A close examination of the event logs showed that the following error was occurring every time I attempted to start the service.
Source: VMware VirtualCenter Server
Event ID: 1000
Description: The description for Event ID ( 1000 ) in Source ( VMware VirtualCenter Server ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to retrieve this description; see Help and Support for details. The following information is part of the event: [Vdb] Failed to obtain database connection. Retrying…
Odd. I checked over the ODBC settings, and everything look correct and tested OK. So not knowing what else to do initially, I decided to put in a Change Management to reboot the server. A few minutes later, our DBA, Marcus, was at my desk.
Marcus indicated that the SQL server that housed the VMware DB was getting repetitive errors in it’s Event Logs that read as:
Event ID: 18456
Description: Login failed for user ‘sa’. [CLIENT: <ip address to VCenter server>]
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
For a lark, I tried starting the VMware VirtualCenter Server service one more time, and sure enough another like error appeared in the SQL Server logs. Well, hell…
Another odd thing… this entry indicates that the SA account is being used for authentication. To the best of my knowledge, our VCenter server should be using a set of custom credentials that Marcus had created for that specific purpose. Another strange fact, Marcus said that the password on the SA was change 2 weeks ago. If that was the cause of the issue, then shouldn’t we have noticed 2 weeks ago?
Scratching my head wasn’t going to solve the problem, so I called up VMware Support (whom I never mind calling because of the top drawer consideration and talent we receive! Thumbs up to VMware for their support staff.) I opened a ticket and asked to be dropped into their live support queue… where I waited for approximately 10 minutes.
Dan came on the line, and had the issue sized up in about 2 minutes. (Like I said, top drawer talent!)
It seems that when the VMware VirtualCenter software was originally installed (by one of my associates), he did not yet have the custom credentials that Marcus had created for us for authentication (and wasn’t patient enough to wait). So during the initial installation, he had used the SA account credentials. Supposedly, the software caches that information and uses it… but it only uses it when the VMware VirtualCenter Server service starts.
WARNING: I am not a VMware Support specialist, and cannot guarantee that the following will resolve a similar issue that you are having, nor will I take responsibility for any data loss you may encounter. I’m merely telling a story about how I found resolution. So use the following information at your own risk… or better yet, call VMware Support!
On the VCenter server, go to the Control Panel and open up the Add/Remove Programs applet. In the list of installed programs, find the VMware VirtualCenter Server entry… and click the [Change] button.
The installation wizard will launch. Step through the wizard, and select to “Repair” the installation. Confirm and accept all of the preselected options… until you get to the “Database Information” window. This is where we found the SA account listed as the username. We entered Marcus’ custom credentials and set off to complete the rest of the wizard.
Then, we encountered an additional complication. During the above mentioned step, we were presented with this error after putting in the custom credentials:
Setup failed to connect to the database pointed to by the given DSN. Please make sure that you have the proper permissions to access your database.
It seems that if it doesn’t have it already, the custom credentials are REQUIRED to have Database Owner privileges on the SQL database when you are going through this wizard. (Afterwards, DBO privileges can be removed from the account, but it requires DBO to complete the setup wizard… and will for any upgrades you may perform to the software in the future.)
One more important point…
Make sure to select [No] when asked if you want to reinitialize your database. Selecting Yes would be a bad thing!
After completing the wizard, I opened up Services and found that the VMware VirtualCenter Server service was once again running.