So fast we had to slow down Windows!

One of dsp's Microsoft SQL Server Consultants talks about private cloud implementation…

“During one of our latest SQL Server 2012 Private Cloud deployments, we came across an interesting problem. We had built a new SQL cloud using some high end technology (Latest HP DL380 G8 and Violin Storage - see below) and were getting ready to deploy our first servers into live. The servers had already undergone a series of performance tests and functionality tests, and had passed with flying colours. No problems at all and configuration was good. The performance tests were pretty staggering – showing the servers were capable of 4X the throughput of the old server (a dedicated HP DL380 G7).

However during pre-production and fail-over testing we encountered a problem when we were simulating a power failure outage on one of the servers.

When the server powered backed up from the simulated failure, it would not re-sync the AlwaysOn Mirror as expected. A little digging found the SQL Server service had not started, even though the service was set to auto start. In the error log we found:

The MSSQLSERVER service was unable to log on as XXXXXXXXX with the currently configured password due to the following error:
There are currently no logon servers available to service the logon request.

To ensure that the service is configured properly, use the Services snap-in in Microsoft Management Console (MMC).

This was quite puzzling.

After a few cups of coffee and some head scratching, we decided to run a debug Windows Start-up with prompt service starts. However this time the problem did not present itself and the server booted normally with the SQL Service started. Hmmmm?

After a little more testing and digging around (and more coffee), we found the issue! The problem was the services were starting so fast, the Network card had not had enough time to obtain an IP address and Network bind to authentic to an Active Directory server.

Setting the SQL Server services to a delayed start resolved the issue, allowing a perfect start every time.

This just left us wondering, why this environment and not others? Seems that the Violin and the fast Gen 8 HP Servers are a wicked combination! The Violin storage is SLC Memory based storage and responds in microseconds (not milliseconds like standard disk storage).

This is means the Violin is 5 times faster to respond to a disk request, than your brain can tell your index finger to move... i.e. pretty darn fast. Given disk is the #1 bottleneck to database servers, it can only be a good thing (now we fixed our start up issue).”