Knowledge

Client Overview:

The client is the world leader in recovery auditing services. Headquartered in Atlanta, GA, the client has over 1000 associates servicing clients in more than 30 countries.

Business Problem:

Following the acquisition of its top competitor, a leading Audit Recovery firm had experienced significant growth in IT assets yet did not standardize or change its purchasing strategy. For several years after the acquisition, systems and supporting gear were purchased on an as-needed basis for each new application or project. The industry began to change and more competition entered the market which resulted in a sharp decline of revenue. In 2006, the business leaders found themselves needing to drastically reduce operating costs as revenue began to decline.

The Facts:

• The IT department was a cost center for the business. Although the company would not be able to function without a strong technology group, the core of the business were the auditors and analysts that provided the billable services to clients.
• The corporate data center had reached capacity and cooling was becoming a major problem.
• The company was nearing capacity at a co-location facility that it paid close to 40k per month for.
• The company had a mature systems management practice which monitored all critical systems and applications.
• The company was in the process of implementing change management practices.
• Virtualization technologies were not being used at the company.

The Solution:

The business asked the CIO to make deep cuts in IT spending, but was not able to provide funding of any kind to address and remedy the problem due to the dire financial state of the company.

Because the two data centers were essentially full and due to the fact that the company had a history of purchasing systems based on application and project use, the most obvious place to begin was with the IT assets themselves. Typically, we find that server assets are highly underutilized and consolidation is almost always possible. This not only lowers the number of physical systems deployed, but can also reclaim significant amounts of expense in maintenance contracts, licensing, backup methods, and engineering support.

We leveraged the already in-place systems management framework to gather and analyze six months worth of performance data for each of the 300 systems spread across all business units. Combining the utilization data with a system interdependency matrix and input from the business owners of the equipment, we classified and created a workload profile for each system in the organization.

We were quickly able to identify close to 100 physical systems within the production environment that were underutilized and in some cases, not being used at all. Further, our workload profiles illustrated that most of the servers purchased within the previous few years were powerful enough to be re-purposed for new projects created by the business to increase revenue. The remaining 200 systems were eliminated from the first phase of the project due to high utilization levels, heavy database activity, and complex interdependencies.

Since the company was not currently using in any virtualization technology, we ran a proof of concept pilot using both VMware Server and Microsoft Virtual Server in order to test and select a platform that could also be used to create a free server farm built with existing equipment in which to consolidate physical servers to.

Using free VMware Server software, we consolidated 30 legacy Windows NT 4.0 systems down to 3 physical blade servers in a single chassis which:

About 60 of the production Windows 2000+ servers were virtualized to about 30 physical servers hosted on Windows 2003 servers running VMware Server software. (Compression ratios are much higher today by using a bare bones hypervisor like VMware ESX server.)

Of the remaining 10 systems, 7 were SQL database servers that served light duty across the enterprise and 3 were web servers hosting about 5 websites in total. We were able to consolidate each function to one instance with plenty of remaining capacity for future growth. Decent savings were realized for future projects from the 6 reclaimed SQL Server licenses alone.

The net remaining 35 physical servers were put into surplus stock and repurposed back to the business throughout the year. We were able to use existing equipment and a newly approved VMware standard to satisfy all 2006 server requests, thus eliminating the need to spend even a single penny on new physical servers that year. The business used most of the reclaimed servers for projects aimed at servicing new revenue streams as it attempted to reclaim the top spot in the Audit Recovery industry.

The Results:

This project was a major success and an excellent example of what is possible for most businesses that have not analyzed and optimized their IT departments recently.

Some highlights include:

• A 500K cost savings related to new hardware and licensing purchases in the IT budget.
• Freed up the rack, power, and port space that 65 physical servers consumed in the data center.
• Eliminated hardware maintenance contracts for the 65 virtualized servers.
• Created a far more efficient way to backup and recover the virtualized systems.
• Actually increased productivity of the Windows NT systems due to the new architecture and faster speeds of the host systems.
• Identified 9 “mystery” systems that were powered on but not in use by anyone within the organization.
• The proven VMware technology allowed IT to finally satisfy a need by developers for QA and development systems without having to loan out physical systems needed for business purposes.

Client Overview:

A major food supplier with customers ranging from small delicatessens to major grocery store chains located throughout the  United States.

Business Problem:

Years of non-standard builds and procedures for implementing technology led to an inefficient infrastructure. Coupled with the fact that the company had a major operations center in the middle of Florida’s hurricane country, new management determined that a new design was needed in 2008.

The company had six regional offices in the Eastern half of the US, relying on local storage and non-technical support staff in most of the sites. Two major data centers in New York and Florida were critical to business, but ran independently of each other causing inefficiencies.

The Facts:

• Government regulations required the company to pay its vendors within ten days of receipt of food product – a disaster could cause the company to shut down if payments were not made on time.
• There was no disaster recovery plan, yet the company’s largest data center was located in the middle hurricane country.
• Most remote sites do not have local IT support and thus support technicians would have to fly to remote sites when routine problems arose.
• Local storage and an aging NAS device was the only storage in use.
• Technical staff was not trained or proficient with VMware, yet had it deployed for some production systems.
• The environment had to be mobile enough to be quickly recovered and to allow for quick migration of systems to another site in the event of an impending hurricane.
• Management created an opportunity to address the entire infrastructure while it began gathering data to create a disaster recovery plan.

The Solution:

Physical servers were assessed and it was quickly determined that most of the 60 critical systems could be virtualized and moved to less than ten hosts spread across the two major sites. Remote sites would get a standardized virtual infrastructure consisting of file, print, database, and application services. Messaging was consolidated to run from the major data center sites.

To meet the requirements for the ability to quickly move running systems to hosts in another region of the US, a new NetApp SAN environment was built. Virtual machines, user data, and backup data was located on SAN units in each location. Each location replicated with the two major data centers and could effectively be locally shut down and kept running elsewhere if necessary.

The Results:

The resulting environment gave the client several things they did not have prior – remote manageability of critical business systems, business continuity and disaster recovery plans, and a shared storage environment.

The remote manageability solved the problem of having to fly technicians out to sites which delayed repair time and was an expensive way to support remote sites. The cutting edge site to site replication and agile nature of the infrastructure gave the business the option of “pressing a button” to begin the short migration of systems from Florida to New York in the event that a hurricane was on it’s way. Locating virtual machines and user data on a SAN that replicated to other sites eliminated the traditional methods of restoring from tape, which would take several days due to the volume of data.