Re: OVH datacenter SBG2 in Strasbourg on fire 🔥

Thu Mar 11 10:37:38 UTC 2021

Sent from my iPhone

> On Mar 10, 2021, at 7:45 AM, Andy Ringsmuth <andy at andyring.com> wrote:
> 
> Sad to see of course, but also a little surprising that fire suppression systems didn’t, well, suppress the fire.
> 
> Unless they didn’t exist?

I am assuming you haven’t had a real datacenter fire before.

I’ve had one fire, seen another, and had an accidental system firing of the gas system.  

In the actual fire, caused by cooling system partial failure, there was no gas and it turned out the system to disable power on sprinkler discharge failed, then the power circuit breakers stayed live despite significant electrical short circuiting in the room as sprinklers fired, and then after 5 minutes the fire department arrived and sprayed water in for over 10 minutes without reducing arcing and fire before building power was successfully disabled and they could put the rest of the fire out.  This one could easily have destroyed its building, also the building Palo Alto Frys was in.

Fun fact: a rack of burning servers displaces sprinkler water around the rack, if it has a top on it, and even if not a vertical stack of burning servers pushes water down the front and back and slightly to the sides of burning servers, not through the systems themselves.

Fun fact: motherboards burn, as does chip encapsulation epoxy and all the wiring, the fan frames, board standoffs, essentially all the RAM and PCI board slots, some capacitors and surface mount devices... hard drives melt and the aluminum casings burn, SSD plastic casings generally burn.  GBICs and other laser diodes smell awful after they catch fire, though it can be hard to tell with everything else that smells.

Fun fact: It does not necessarily take many burning servers to put the room integrity at risk, even with sprinklers going and the fire department spraying water in. 

Fun fact: All the servers that get wet but don’t burn will rust.  And everything in the room near the sprinklers that are going off will get wet.  This is very sad as you watch many millions of dollars of brand new 42U racks prestuffed with HP enterprise servers oxidizing away while you wait for the garbage truck.

Fun fact: Combustion soot is conductive and even things that didn’t get wet probably are dead.

Fun fact: late era Sun Microsystems server boxes were very nice waxed cardboard, very well made, apparently fire resistant more than anyone would have thought but that were also water resistant enough that you’d have a new in box server submerged in a box still full to the brim with water days later.

Fun engineering advice: The window for critical data recovery from hard drives that are visibly corroding from water damage short of immersion is probably 48-72 hours, but run them outside any casing on a fire resistant table and have all of CO2 and dry chemical fire extinguishers ready... and a fire hose, and handy building fire alarm.  Corroding water damaged SSDs are lower power draw and somewhat less likely to start another fire, but take the same precautions.

The fire I saw but that wasn’t mine burned a building with halon and pre action sprinklers more or less to the steel columns and roof beams, despite the fire department arriving in 5 minutes and trying to actively suppress.  Not enough openings for them to safely get to the fire before it was out of control, and not enough water flow available in all the sprinklers once it took off.

That one had evidently burned aluminum rack posts by the time it was over, not just melted them...

Systems and datacenters aren’t built to eliminate fire risk.  They just aren’t.  You can contain or control most office fires with sprinklers, and certainly evacuate.  Datacenters with emergency power batteries in the envelope often have enough stored energy to set the room on fire despite sprinklers.  If AC cutoffs fail and circuit breakers don’t trip mains power will as well.  Too many systems and storage and networking hardware components can burn.  

-george

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nanog.org/pipermail/nanog/attachments/20210311/a68f17f1/attachment.html>