What to expect after a cooling failure

Erik Levinson erik.levinson at uberflip.com
Wed Jul 10 03:50:58 UTC 2013

Thanks. I should also mention that most of the gear was still on but we had turned off many VMs on physical servers within the first 2.5 hours, so the CPU and hard drive / io load was around zero on such servers. Most of the servers in the hotter suite had fans running at over 75% vs. about 35% in the cooler suite and ambient temp was down to 32 degrees Celcius within four hours.

Erik Levinson
CTO, Uberflip
1183 King Street West, Suite 100
Toronto ON  M6K 3C5

-----Original Message-----
From: "Bryan Tong" <contact at nullivex.com>
Sent: Tuesday, July 9, 2013 11:42pm
To: "Erik Levinson" <erik.levinson at uberflip.com>
Cc: "NANOG mailing list" <nanog at nanog.org>
Subject: Re: What to expect after a cooling failure


In my experience with heating issues the only thing that really degrades
quickly in event of overheating are hard drives. If you had them spun down
it should be fine.

CPU / Memory / Motherboards will be fine.

The only other thing I can think of having possible issues are PSU's but if
they were powered off should be fine as well. Maybe melted wires but I dont
think it was hot enough for that.


On Tue, Jul 9, 2013 at 9:28 PM, Erik Levinson <erik.levinson at uberflip.com>wrote:

> As some may know, yesterday 151 Front St suffered a cooling failure after
> Enwave's facilities were flooded.
> One of the suites that we're in recovered quickly but the other took much
> longer and some of our gear shutdown automatically due to overheating. We
> shut down remotely many redundant and non-essential systems in the hotter
> suite, and transferred remotely some others to the cooler suite, to ensure
> that we had a minimum of all core systems running in the hotter suite. We
> waited until the temperatures returned to normal, and brought everything
> back online. The entire event lasted from approx 18:45 until 01:15.
> Apparently ambient temperature was above 43 degrees Celcius at one point on
> the cool side of cabinets in the hotter suite.
> For those who have gone through such events in the past, what can one
> expect in terms of long-term impact...should we expect some premature
> component failures? Does anyone have any stats to share?
> Thanks
> --
> Erik Levinson
> CTO, Uberflip
> 416-900-3830
> 1183 King Street West, Suite 100
> Toronto ON  M6K 3C5
> www.uberflip.com

Bryan Tong
Nullivex LLC | eSited LLC
(507) 298-1624

More information about the NANOG mailing list