Off the grid: protecting mission critical data centres

North American’s power infrastructure is vulnerable to outages because of age, and disruptions caused by nature or human interference. PHOTO: THINKSTOCK

I took my first post 9/11 flight, travelling from Toronto to Los Angeles on business a few weeks after the terror attacks. From my window seat looking down from 33,000 feet through cloudless skies, I realized just how much our lives had changed. I couldn’t help but note bridges in the middle of nowhere, dams that looked untended and mile after mile of unguarded power lines crisscrossing the landscape. The view I found so serenely beautiful in the past had become unsettling.

Future attacks could cripple segments of the infrastructure for extended periods, which would be massively disruptive and cause grave economic damage. Included in this havoc would be the impact on the mission critical data centres with limited back-up power that control so many essential processes and services. One way to keep critical infrastructure operating would involve taking these data centres off the grid.

How acute is the risk? As low profile, defenceless “soft targets” go, America’s electric grid is among the softest, owing mostly to its age. Outages have increased at a rate of 10% per year since the 1990s and more than 3,600 occurred nationwide last year averaging 47 minutes. Canada’s power infrastructure is also feeling its age and is due for almost $300 billion in investment between 2016 and 2030, according to a 2011 Conference Board of Canada report. Indeed, power infrastructure in both countries is susceptible to modern threats from physical or cyber attacks, as well as changing weather patterns.

First the human threats. According to recent analysis in USA Today, the US power grid is affected once every four days by outages caused by either a cyber attack involving a virus or hacking; or a physical attack like the one involving a man in Scott, Ark. He climbed a tower and using only a wrench, disconnected a power line and placed it on a nearby railway track. This resulted in the destruction of an electrical station, causing $4 million in damage.

Meanwhile, outages caused by extreme weather are becoming more frequent and repairing the catastrophic damage can be complex, time consuming and expensive.

Who can forget the ice storms that swept through Central Canada and the Northeast US in 1998 and 2013 that caused such extensive damage. Power was restored to some areas within hours, but many others were without electricity for days, and in some cases weeks.

Forest fires in the Western US and Canada, partly the result of climate change, also join the list of potential threats. Without actually burying cables underground there is no way to safeguard transmission lines.

Data centres that perform mission critical functions are more complex and costly to build, and they’re designed to maintain an uptime standard because of the high cost of being offline, even for a few minutes a year. Banks, credit card companies and government services fall into this category. Other mission critical applications involve keeping critical infrastructure systems and the services they control operating.

But data centres without power, whether or not they have hacker-proof software and unbreakable encryption systems, are of no use.

Operators have put comprehensive disaster recovery plans in place to minimize disruption. Sites are chosen, preferably, in locations where power from different electrical grids can be accessed for redundancy. “A” and “B” power is distributed within the site using isolated redundant switchgear and separate electrical distribution networks. And large, uninterruptable power supplies (UPS) are installed to deliver instantaneous bridge power during an outage. The UPS is backed up by batteries or by kinetic energy storage devices. At the end of this power chain are large, diesel generators that engage when electricity from the grid is lost for more than a minute or two. Onsite backup generators are usually provisioned with enough fuel onsite for anywhere from 48 to 72 hours of operation at full power.

My colleague Paul De Groot and I began researching the use of alternative energy sources last year as part of a project to develop an advanced, ultra-efficient design for a modular Net Zero data centre on behalf of a large Silicon Valley corporation. Net Zero Energy Buildings is a relatively new “green” standard that incorporates a standalone energy source powered by renewable fuel. Power produced by these facilities is supplied back to the grid in equal measure to power consumed.

Universal design

Our objective was to provide a universal base design that could be replicated, with minor tweaks, in a variety of locations.

We proposed a fully modular design, with components built in a factory and shipped to site 95% complete, positioned on pre-poured concrete pads and connected to electrical and mechanical systems. Modular data centres use space more efficiently. They can be built in half the time of traditionally constructed infrastructure and cost approximately 30% less.

As modules are added to the base design and installed over time, clients scale the data centre to match current needs. This keeps capital cost down and assures the most productive use of the money spent on infrastructure. Assets are written down over seven years (as are other IT assets) versus the 30-year standard for conventional bricks and mortar buildings.

Outmoded cooling systems and poor space utilization are the leading causes of data centre inefficiency. Emerging guidelines from industry groups such as ASHRAE are encouraging the use of warmer air to cool data processing equipment more efficiently by bypassing the chiller. Cooling towers facilitate convection cooling using outside air. In this mode, the only energy consumed is by fans moving air inside the modules and pumps cycling cooling water through the system. Compact closed loop water-cooling systems enable much higher equipment density per rack without relying on the oversized aisles required in ambient-cooled data centres. This allows for a substantial increase in efficiency while reducing the footprint of the white space by as much as 40%.

Optimal technology

Several renewable power-generating systems were considered but the most common ones were in one way or another impractical. A solar array would require 1.6 acres of land in a sunny state (where land is cheap) to house enough panels to generate 900 kilowatts of power. Intermittent energy from wind power is even less practical in urban areas where data centres are built to be close to users, power grids and fibre cable trunks. And advanced combustion engine systems that use biofuel must contend with inconsistent availability, as well as engines that are too big to achieve the needed scalability.

The optimal technology turned out to be solid oxide fuel cells manufactured by Bloom Energy, a Northern California green energy company. Its energy cells convert fuel, typically natural gas, into electricity through a clean electro-chemical process, without combustion. An individual Bloom energy server provides 250 kilowatts of power and requires about one tenth of the space required for a solar array. More energy servers can be added as demands increase, which avoids unnecessary first cost.

With lower natural gas prices thanks to new sources of domestic supply, Bloom’s systems are now competitive in regions where utility rates are at about 10 cents per kilowatt hour or higher.

Bloom Energy’s cells used in conjunction with closed loop water-cooling as the primary source of power generation eliminates traditional power protection, back-up power and energy storage systems. This reduces the cost of construction and simplifies system maintenance. Other benefits include: power produced without transmission losses, which can net an additional 5% to 6% in efficiency; no diesel generators so no permits, emissions testing and noise abatement; no onsite bulk storage tanks; no UPS and power losses drop by an additional 2% to 8%; no air-conditioned battery rooms; and the cells don’t consume water or produce emissions.

The technology’s key advantage is its use of natural gas, delivered via underground distribution systems that are not as vulnerable as power lines to extreme weather or sabotage. If needed, onsite (above ground) storage of compressed natural gas is simpler and more economical than diesel fuel stored in large underground tanks.

Major outages caused by weather related catastrophes can take grid power offline for weeks depending on the severity of the damage, and targeted attacks on grid infrastructure could disrupt power for even longer. Yet current mission critical data centres only operate for up to three days using fuel stored onsite.

If we are to weather all manner of coming storms, critical IT assets must be sheltered from a vulnerable power grid. Building data centres with independent energy sources is worthy of serious consideration.

Peter Jeffery developed and patented the first high efficiency closed loop, water-cooled enclosure for mission critical applications. He’s a co-founder of DataThermic Infrastructure Solutions, a designer of modular infrastructure systems for data centres with offices in Toronto and Silicon Valley, Calif.

This article appears in the September 2016 issue of PLANT.