"Hypothetical" Datacenter Overheating

Mon Jan 15 14:23:37 UTC 2024

At 09:08 AM 2024-01-15, Mike Hammett wrote:
>Let's say that hypothetically, a datacenter you're in had a cooling 
>failure and escalated to an average of 120 degrees before 
>mitigations started having an effect. What are normal QA procedures 
>on your behalf? What is the facility likely to be doing? 
>What  should be expected in the aftermath?

One would hope they would have had disaster recovery plans to bring 
in outside cold air, and have executed on it quickly, rather than 
hoping the chillers got repaired.

All our owned facilities have large outside air intakes, automatic 
dampers and air mixing chambers in case of mechanical cooling 
failure, because cooling systems are often not designed to run well 
in extreme cold.  All of these can be manually run incase of controls 
failure, but people tell me I'm a little obsessive over backup plans 
for backup plans.

You will start to see premature failure of equipment over the coming 
weeks/months/years.

Coincidentally, we have some gear in a data centre in the Chicago 
area that is experiencing that sort of issue right now... :-(