Google, Oracle datacenters melt down in extreme European heatwave

A punishing heat wave currently gripping the UK and other parts of Europe is wreaking havoc on Google and Oracle cloud servers, particularly those located in datacenters that are not built to take the high heat. Over 34 locations exceeded the UK's previous temperature record of 38.7°C recorded in July 2019, and the country recorded its hottest-ever temperature, 40.3°C (104.5°F) in Coningsby in Lincolnshire.

As reported by The Register, Oracle has been forced to power down some hardware, which could leave some customers unable to access some Oracle Cloud Infrastructure services, while Google is reporting "experiencing elevated error rates, latencies or service unavailability" across multiple cloud services in Europe.

In both cases, the trouble was caused by a breakdown of cooling systems that were struggling to cope with the extreme heat. "As a result of unseasonal temperatures in the region, a subset of cooling infrastructure within the UK South (London) Datacentre has experienced an issue," Oracle's system status website says. "The relevant service teams have been engaged and are working to restore the affected infrastructure back to a healthy state. Our engineers expect redundancy to the impacted cooling infrastructure to be restored within the next 1-2 hours, after which services will begin to be recovered."

In an update posted at 3 pm ET, Oracle said that work on cooling systems is continuing, and temperatures are dropping as a result of both the repair work and the shutdown of "non-critical" systems. "As the operating temperatures approach workable levels, some services may start to see recovery."

Google also reported a cooling failure earlier today impacting its europe-west2 region. "This caused a partial failure of capacity in that zone, leading to VM terminations and a loss of machines for a small set of our customers," a Service Health message states. "We’re working hard to get the cooling back on-line and create capacity in that zone. We do not anticipate further impact in zone europe-west2-a and currently running VMs should not be impacted. A small percentage of replicated Persistent Disk devices are running in single redundant mode.

"In order to prevent damage to machines and an extended outage, we have powered down part of the zone and are limiting GCE preemptible launches. We are working to restore redundancy for any remaining impacted replicated Persistent Disk devices."

While Google and Oracle struggle to get their systems fully online, a Bloomberg report says that some data centers in the UK are taking a more basic approach to cooling by spraying their roof-mounted AC units with water, which lowers the ambient temperature around the coils and enables the units to continue to function efficiently. It's a short-term solution—Sophia Flucker, director at consulting firm Operational Intelligence, said using untreated hard water could lead to a scale buildup on the equipment and possibly shorten its life—but since there are clearly no long-term solutions to any of this on the horizon, blasting untreated water all over sensitive, expensive AC systems is what we have left. 

Google actually has a deal in place that grants it access to millions of gallons of groundwater per day for cooling purposes, which doesn't seem ecologically ideal either, but that's only in the US—this sort of extreme heat presumably wasn't expected to ever happen in the traditionally much-cooler UK.

The UK and much of Western Europe are struggling with a punishing heatwave that has also caused a surge of fires across London and forced the Royal Air Force to halt flights in and out of one of its bases. Large fires burning across Spain, France, Portugal, and Greece have also destroyed swaths of vegetation and forced thousands of people out of their homes.