We have been alerted to a disk failure in both PEMVZMPS20 and PEMVZMPS23. Due to it's RAID configuration no data loss has occured of course.
To get the RAID back up and running to it's full potential we are going to replace the failed disks tonight.
When: 3rd of March 2010 at 21:00 (Expected downtime 15 minutes per node.)
Affected nodes are:
81.17.254.85 pemlinweb04.blacknight.com
81.17.254.86 pemlinweb05.blacknight.com
81.17.254.45 mysql71.cp.blacknight.com
We will update this blog post once completed.
Update 21:32 - PEMVZMPS23 has been completed, both pemlinweb04/05 are back online. We are waiting on a disk check to finish on PEMVZMPS20, once complete the node will be brought online. ETA 15 minutes.
Update: 22:20 - both nodes are back up and the services are restored. As the raid arrays are rebuilding thing sill be a little sluggish until it's completed. This will take several hours.
Summary: Tonight Monday March 1st we're moving the Provisioning node known as OSSCORE to new hardware. This is due to the sheer volume of customers going into the system on a daily basis. This has caused our provisioning back end to get slower and slower which is causing cp.blacknight.com (web hosting) to slow down.
When: March 1st 2010 starting at 22:00 until 01:00 on March 2nd.
What: From 22:00 hours we'll be working on the migration, the cp on cp.blacknight.com will be turned off for the duration of this migration in order to prevent inconsistencies
Services affected: Management of your web hosting plans, e-mail, databases etc will not be available during this window but your hosting will be unaffected.
Update 23:00: this window has been completed successfully a full 2 hours early.
Ragnell is currently having issues which is causing it to become unresponsive. There is an engineer on site getting it backup, and it should reappear within the next 10mins.
We are currently experiencing issues with our shared hosting server Ragnell. Our engineers are working on resolving this currently.
Update 5:58PM This has been resolved.
We are currently experiencing issues with our shared hosting server Bors (81.17.252.40)
Our engineers are working on resolving this currently and will update this post once more information is available.
Update 6:52PM: This issue has been resolved.
At around 15:30 balin.blacknight.ie stopped responding to requests. As we had an engineer on site who also couldn't log in locally, we rebooted it straight away and it was back up by 15:40.
While we don't know exactly what caused the issue yet, it looks like server was run out of memory, possibly due to a massive surge of queries to MySQL.
The server Bediver (81.17.248.30) needs to be reboot urgently.
Our engineers are doing this now and the downtime will be no more than 5 minutes.
UPDATE: This is complete
Pemwinweb09(81.17.250.41) is currently experiencing issues. We're currently looking into it and will update here as soon as we know more.
UPDATE 17:38: The server is up and running again and we're currently investigating to find the cause of the outage.
There will be an upgrade of the PHP versions on Gorlois, Priamus and Rivalin to PHP 5.2.12 this evening in order to fully sort an issue we've been having with Installatron. There should be only a few seconds downtime required as Apache restarts with the new version.
UPDATE: 18:10
All three servers have been successfully updated. Gorlois' had slightly more downtime than expected as an automated process restarted Apache in the background while PHP was being updated. This caused PHP to stop working for about 2 minutes.
UPDATE: 08:52 Tuesday Feb 2nd
As mentioned above the server Gorlois had some additional knock on problems after this upgrade. During the compile of PHP exim decided that it wouldn't accept e-mail from outside. This wasn't caught until the alerting system sent it's first reminder about the service being down some time after. Gorlois' SMTP service came back at 20:17 last night and people were able to send e-mail again and e-mail from outside started to flow. We don't believe any e-mail was lost during this window.