We are currently experiencing issues with our shared hosting server Ragnell. Our engineers are working on resolving this currently.

Update 5:58PM This has been resolved.

Due to increased traffic we are seeing on one of our MySQL nodes we are going to migrate it to it's own dedicated hardware.

The downtime will occur tonight at 22:00, Feb 8th 2010

The downtime for the affected node will be no more than 45 minutes.

The affected node is:

mysql106.cp.blacknight.com
mysql106int.cp.blacknight.com

We will update this post once completed.


Update: This work has now been completed.

We are currently experiencing issues with our shared hosting server Bors (81.17.252.40)

Our engineers are working on resolving this currently and will update this post once more information is available.

Update 6:52PM: This issue has been resolved.

Due to a software error we are seeing on the hardware node, PEMVZWIN02, we need to reboot this node asap.

The node will be down for no more than 10 minutes per VPS on the server.

The affected VPSs are:

78.153.208.116  VPS-238
78.153.208.123  VPS-258
78.153.208.16   VPS-262
78.153.208.127  VPS-282
78.153.208.28   VPS-287
78.153.209.211  VPS-288
78.153.208.154  VPS-317
78.153.208.168  VPS-331
78.153.208.149  VPS-337
78.153.208.170  VPS-341
78.153.208.172  VPS-342
78.153.208.169  VPS
78.153.208.171  VPS-344
78.153.209.151  VPS-346
78.153.208.15   VPS-347
78.153.208.62   VPS-353
78.153.208.176  VPS-354
78.153.208.177  VPS-355
78.153.208.179  VPS-357
78.153.208.184  VPS-362
78.153.210.75   VPS-387
78.153.208.205  VPS-390
78.153.208.218  VPS-407
78.153.208.222  VPS-409
78.153.208.223  VPS-410
78.153.209.118  VPS-620
78.153.209.160  VPS-664
78.153.209.164  VPS-667
78.153.209.176  PARTYCENTRAL
78.153.209.107  VPS-710

We will update this post once completed.

Update 12:57: This server has been rebooted now. We are now bringing all VEs back online

Update 13:17: All services are resumed.

At around 15:30 balin.blacknight.ie stopped responding to requests. As we had an engineer on site who also couldn't log in locally, we rebooted it straight away and it was back up by 15:40.

While we don't know exactly what caused the issue yet, it looks like server was run out of memory, possibly due to a massive surge of queries to MySQL.

The server Bediver (81.17.248.30) needs to be reboot urgently.

Our engineers are doing this now and the downtime will be no more than 5 minutes.

UPDATE: This is complete

Pemwinweb09(81.17.250.41) is currently experiencing issues. We're currently looking into it and will update here as soon as we know more.

UPDATE 17:38: The server is up and running again and we're currently investigating to find the cause of the outage.

There will be an upgrade of the PHP versions on Gorlois, Priamus and Rivalin to PHP 5.2.12 this evening in order to fully sort an issue we've been having with Installatron. There should be only a few seconds downtime required as Apache restarts with the new version.

UPDATE: 18:10

All three servers have been successfully updated. Gorlois' had slightly more downtime than expected as an automated process restarted Apache in the background while PHP was being updated. This caused PHP to stop working for about 2 minutes.

UPDATE: 08:52 Tuesday Feb 2nd

As mentioned above the server Gorlois had some additional knock on problems after this upgrade. During the compile of PHP exim decided that it wouldn't accept e-mail from outside. This wasn't caught until the alerting system sent it's first reminder about the service being down some time after. Gorlois' SMTP service came back at 20:17 last night and people were able to send e-mail again and e-mail from outside started to flow. We don't believe any e-mail was lost during this window.

We experienced some issues with one of our mysql nodes at 19:35. Some users may have seen connection timeout errors on their database connections for a peroid of 3/5 minutes.

The affected node was: mysql71.cp.blacknight.com

This has been resolved now and normal service is restored.

Due to a hardware upgrade we are migrating the MySQL node listed below to new hardware this evening. All details are listed below

When?
The downtime will occur tonight at 22:00 hours, 27th of January 2010

What will be offline?
Any databases located on these servers:
mysql360.cp.blacknight.com
mysql360int.cp.blacknight.com

For how long?
The downtime will last an hour.

We will update this post once completed.

Update 23:00

This update completed at 22:24. This mysql server node is now live on a new hw node and is already showing better performance for customers using it.