French cloud operator OVH is yet to bring all customers affected by its Strasbourg data centre fire back online, and the company’s CFO has described ongoing restoration efforts as a “real nightmare.”
The fire took place on 10 March and destroyed the “SBG2” hall of the Strasbourg data centre, damaged SBG1 so badly it won’t be revived, and led to a massive effort to clean salvageable kit so it could be installed in the remaining three data centres at Strasbourg, or moved to other OVH facilities.
Work is going slowly, according to the most recent tweet from OVH co-founder and chair Octave Klaba.
Update April,14 9pm
SBG1/E > Croix > SBG3
new 200 servers were shipped to Floor 5 which is now full. Tomorrow, we start filling Floor 3. Step by step. Yes, it’s slow. Real nightmare. pic.twitter.com/ylzMjzlEY3
— Octave Klaba (@olesovhcom) April 13, 2021
In other tweets, Klaba said that all servers due to be moved from SBG1 to SBG4 have made the trip, with moves from SBG1 to SBG3 due this week.
“All servers are cleaned and tested, but it takes time to reassemble the racks… a huge Tetris,” he wrote.
OVH no longer has a link to its incident report on the front page of its website, but a 12 April update revealed the following progress towards restoration:
- 92 per cent of servers for OVH Bare Metal Cloud have been delivered to customers
- 86 per cent of customer services running on Public Cloud instances have been replaced
- 99 per cent of Public Cloud Storage has been replaced
- 79 per cent of Hosted Private Clouds have been restored
- 84 per cent of Bare Metal Cloud Virtual Private Servers have been restored
The new update said cleaning each rack prior to reinstallation in a new data centre takes seven hours, but that OVH staff are getting faster every day. However, France’s renewed COVID-19 lockdown is not helping.
“We have had to source travel, accommodation and catering for all our employees who have come in to assist on site, while respecting government protocols, which of course made the operations more complex,” the company said.
OVH says burned data centre’s UPS, batteries, fuses in the hands of insurers and police
Pandemic-related challenges, and the sheer complexity of the restoration effort, mean OVH has not offered a resolution date for the incident.
The company is also working on vouchers customers can use to pay for services and will not bill for services affected by the fire or replacement services ordered by customers. ®