Noise Filter: New York Data Center Employees Face Hurricane Sandy Challenges
Every now and then, an exciting or controversial issue triggers a flood of online discourse. For our Noise Filter feature, the WHIR pans the raging rivers of opinion for shining nuggets of useful commentary.
As New York continues its recovery effort in the aftermath of Hurricane Sandy, data center operations, for the most part, are stable.
While Hurricane Sandy posed a lot of challenges for data centers in lower Manhattan, including issues around data center flooding and generator fuel shortages for PEER 1 and Internap’s 75 Broad Street facility, staffing also posed a tremendous challenge.
With fuel shortages, subway closures, and insufficient food and water, data center providers had to be creative in getting their staff on location and protect their safety.
For example, when Internap evacuated its data center on Monday, no remote hands support was available to assist in customer equipment shutdown. On Ycombinator, Internap data center customers seemed understanding of the fact that “life safety” was its number one priority.
People’s lives are at stake. If the service is important enough that you’d expect them to stay behind and keep it running for you, it is probably important enough that it exists in multiple, geographically separated data centers.
By mid-week, around 25 employees of PEER 1, Fog Creek Software and SquareSpace were on hand to carry fuel to the generator on the 17th floor of the data center since the pump malfunctioned. Employees, as well as additional help from Queens and Brooklyn, worked in ‘bucket brigade’ shifts to fuel the data center.
We have a big crew teaming up tonight for a midnight bucket brigade, and another in the morning. Traffic is terrible with gridlock everywhere and no public transport, so we are scheduling carefully.
The fuel pump was installed and started working on Friday, according to the Squarespace status blog. While the ‘bucket brigade’ showed incredible dedication in keeping the data center running, Patrick Thibodeau at ComputerWorld wondered if the efforts were worth the risk.
The building had no generator power. No utility power. What was the condition of the building fire suppression systems? Was the fire alarm system operational? Were fire doors kept open as fuel was moved up the stairs? What if a fuel bucket had been dropped and fuel spilled down the steps? If this was my business, servers and job, my personal inclination would have been to help haul that fuel up. I totally get that. But there’s also an obligation to step back and look closely at whether the actions taken here were prudent and wise…Perhaps there is a margin of safety involved here that I’m unaware of, but these are questions that will have to be asked because this can’t happen again.
A third of a mile away from 75 Broad Street is New York Internet’s New York City data center, which was “out of reach of the storm surge” and continued operating throughout Hurricane Sandy without any outages, according to a Netcraft post.
Demand for fuel is very high in the areas affected by the storm. Phillip Koblence, founder of NYI, said that poor availability of gasoline for staff to get to and from data centers was his worst problem, followed by sleeping and washing facilities. Queues to get gasoline were reportedly two miles long, and vehicles containing fewer than 3 people were not being allowed into Manhattan between 6 am and midnight.
Koblence told Netcraft that he had no power or water at home, and difficulties getting food anywhere near work were also adding to the problems. NYI expected commercial power to be shut off as the storm hit, and its New Jersey and New York City data centers have been running on diesel generators since Monday.
As teamwork continues to be essential in the Hurricane Sandy recovery efforts, Dan Ushman, co-founder and vice president of web host SingleHop, made a public call to web hosting companies to help New York businesses get back online.
We are extending an open invitation to the rest of the tech community. If anyone wants to join us, by providing hosting, IT services, development… By doing this, we can pool resources (people, infrastructure) and thereby celebrate the American tradition of coming together in times of crisis. We invite any hosting company or individual with expertise, resources or just a desire to learn, to join us in leveraging the cloud to help those who have been impacted by the disaster; either directly (infrastructure providers whose data centers are flooded and in need of temporary data relocation), or indirectly (those who are relying on impacted services and need alternatives to communicate with colleagues, family and friends).
Talk back: Have you or your team helped New York City area data centers in their disaster recovery efforts? How were you impacted by Hurricane Sandy, and what were some of the challenges you faced around staffing and data center availability? Let us know in a comment.[Source: Noise Filter: New York Data Center Employees Face Hurricane Sandy Challenges]