One of the RAID disks in the database server 5 failed and it needs to be replaced as soon as possible.
I requested the replacement and expect it to be done in the next couple of hours. This requires the server to go offline but it should take less than an hour.
I will update here once the intervention starts/finishes.
Only instances using the Database 5 will be impacted (see the database server in your Masto.host account).
Sorry for the inconvenience. This week servers decided to act up :|
There is a problem with the disk replacement that is causing the server not to boot. They are trying to understand what could be causing the problem.
I am really sorry but it's something that I have no access to and will have to rely on the information I am provided, causing me to provide timings and have expectations that are inaccurate.
I'm contacting everyone I can to obtain more informations and ways to speed up the process.
Everything is looking good with Database server 5 and everyone is back online.
Still, the faulty drive is on the server and I still need to deal with that because running with a single disk RAID is dangerous. I do have remote backups that run daily but only expect to use them as a last case scenario.
Will spin a test server to see if I can replicate the issue and see if I can work around it.
Tomorrow will provide more information.
Enormous apologies to anyone affected by this downtime.
@mastohost If they can't sort that they should provision you a spare server to migrate to, it's not like someone the size of OVH doesn't have entire spare servers laying about.
@Dragon It's one of the options on the table, I just don't really want to be again in the same situation, with a server down for hours. I have done disk replacements with them in the past without any major issues, around 30 minutes and it's back online. Hope to have a plan in place in the next hours.
After much back and forth, I have decided to migrate the databases on database server 5 one by one to a new server. It's the safest option to be sure there is no endless downtime, where I am left waiting without any information and ways to act.
After the migration of all data is done, I will play with the server to make sure future disk replacements don't cause the same situation.
This means that there will be some downtime during the database migrations but for small databases it will be minimal, a couple of minutes, but for larger ones it can mean up to two hours of downtime. I will email the larger instance admins to inform of a specific downtime for their instance.
All databases have been migrated from Database Server 5 🎉
This issue will not cause any more problems.
Next week, me and the data center team will be doing some investigation into Database Server 5 to determine the best way to avoid a problem like this from happening again.
Terribly sorry for the inconvenience this caused and any issues or questions, please let me know.
Massive SNAFU! I can't apologise enough but it was not on my end.
The old drive stored the EFI and that is required to boot the server. The person that did the disk replacement didn't know that (EFI cannot be on the RAID), so they were hours attempting to reboot the server without EFI, until they gave up and handed me the broken server.
Now, an attempt will be made to place the old drive back and see if the server boots like that and then I can fix the situation.
Add comment