We are currently experiencing issues with our linux shared hosting node: balin
Our engineers are working on resolving this asap. This blog post will be updated once resolved.
Update: 16:47: This issue has now been resolved.
We have been alerted to a disk failure in both PEMVZMPS20 and PEMVZMPS23. Due to it's RAID configuration no data loss has occured of course.
To get the RAID back up and running to it's full potential we are going to replace the failed disks tonight.
When: 3rd of March 2010 at 21:00 (Expected downtime 15 minutes per node.)
Affected nodes are:
81.17.254.85 pemlinweb04.blacknight.com
81.17.254.86 pemlinweb05.blacknight.com
81.17.254.45 mysql71.cp.blacknight.com
We will update this blog post once completed.
Update 21:32 - PEMVZMPS23 has been completed, both pemlinweb04/05 are back online. We are waiting on a disk check to finish on PEMVZMPS20, once complete the node will be brought online. ETA 15 minutes.
Update: 22:20 - both nodes are back up and the services are restored. As the raid arrays are rebuilding thing sill be a little sluggish until it's completed. This will take several hours.
Summary: Tonight Monday March 1st we're moving the Provisioning node known as OSSCORE to new hardware. This is due to the sheer volume of customers going into the system on a daily basis. This has caused our provisioning back end to get slower and slower which is causing cp.blacknight.com (web hosting) to slow down.
When: March 1st 2010 starting at 22:00 until 01:00 on March 2nd.
What: From 22:00 hours we'll be working on the migration, the cp on cp.blacknight.com will be turned off for the duration of this migration in order to prevent inconsistencies
Services affected: Management of your web hosting plans, e-mail, databases etc will not be available during this window but your hosting will be unaffected.
Update 23:00: this window has been completed successfully a full 2 hours early.
Ragnell is currently having issues which is causing it to become unresponsive. There is an engineer on site getting it backup, and it should reappear within the next 10mins.
We are currently experiencing issues with our shared hosting server Ragnell. Our engineers are working on resolving this currently.
Update 5:58PM This has been resolved.
We are currently experiencing issues with our shared hosting server Bors (81.17.252.40)
Our engineers are working on resolving this currently and will update this post once more information is available.
Update 6:52PM: This issue has been resolved.
At around 15:30 balin.blacknight.ie stopped responding to requests. As we had an engineer on site who also couldn't log in locally, we rebooted it straight away and it was back up by 15:40.
While we don't know exactly what caused the issue yet, it looks like server was run out of memory, possibly due to a massive surge of queries to MySQL.
