Recently in Database Category

When: Monday August 16th from 02:00 until 8am

What: Control panel software, provisioning system, Agents on all hardware nodes, mysql nodes, web servers will all be upgraded from version 2.9.4 to version 5.0. During this window access to the control panel will be restricted. However e-mail, hosting etc services should not be affected by this upgrade.

Changes made to webspaces, new or existing e-mail accounts added or modified, new database creations etc during this window would be ill advised.

Services Affected:

cp.blacknight.com - Control Panel only.

Windows Shared hosting servers may be restarted. only if the upgrade of the management application proves to be problematic.

Linux Shared hosting servers may be restarted. only if the upgrade of the management application proves to be problematic.

Exchange Servers may be restarted, but due to the design of the Hosted Exchange no downtime should be noticed.

Qmail servers will have their imap/pop3/smtp services restarted as new versions of the software get put in place. Also changes may be made to LDAP so there could be intermittent authentication issues.

Sitebuilder servers should not be affected.

VPS nodes will have some software upgraded, but end users should experience no downtime.

Domain registrations and modifications will not be affected by this upgrade.

Update 07:40 Monday 16th:

The upgrade is still being performed at the moment, but it's in it's final stages. We'll post an "all clear" notice when it's complete.

Update 08:45 Monday 16th:

During the upgrade Parallels appear to have broken the provisioning system somehow. As this drives the Control Panel the CP is still down. They're working to resolve this issue but we've got no ETA as of yet.

Update 1045
The maintenance and upgrade has been completed. If anyone has any specific issues post-upgrade please contact our helpdesk

Our engineers are currently seeing issues with one of our older Direct Admin servers:

Ector.blacknight.ie - 81.17.252.50

The server was rebooted not long ago and there are some lingering issues that may affect the receiving of email and possibly cause downtime on your websites.  We are working on this issue at the moment and our highest priority is to get this server back up and running as soon as possible.

We also hope to implement some changes on this server over the upcoming week to resolve the intermittant issues this server has been having over the last couple of weeks.


Update 12:50 - The server is back up now and our engineers are checking all services to ensure they are up and running.  They are still working on the server to restore all back to a normal level of service.

Update 15:10 - I'm afraid the server has gone down again and our engineers are working on restoring service once more.  We are still investigating the root cause of these issues today.

Update 15:26 - The server is still a little slow but services are back now.  We will be contacting some of the busier sites on this server in order to alleviate the load issues.

Update Aug 5th 11:22 - This server has had some config changes, we've moved a few of the busier wordpress blog sites off it and also we've found a problem in the AntiSpam system that was causing lookups on dead realtime blacklists. All of these changes appear to have resolved the problems we were seeing on this server for the past couple of weeks. We'll be monitoring it closely for the next week and if there are no issues in that timeframe we'll consider this issue closed once and for all.

We were experiencing issues with the MySQL database node: mysql71.cp.blacknight.com.

Our technical team have been working on an issue with one of the MySQL server nodes which has been giving intermittent issues over the last few hours.

You can check the current server status here

Update @ 22:00 : This server has been stable for several hours at this stage, but we're continuing to keep an eye on it.

We've noticed increased utilization on some VEs located on the hardware node PEMVZMPS14. As a result we would like to perform a memory upgrade on the server.

The downtime for the node will be minimal at aprox 10/15 minutes.

The affected nodes are MySQL nodes and are as follows:

mysql242.cp.blacknight.com
mysql245.cp.blacknight.com

The downtime will occur on 25th of May 2010 at 08:00AM
UPDATE: This has been rescheduled to the 26th of May 2010 at 8:00AM

Thank you for your understanding and we will update this blog post as soon as the work is completed.

UPDATE 08:07AM: This has now been completed.

A hard disk has failed on our shared hosting hardware node PEMVZMPS35.

No data loss has occurred due to our RAID configuration on the server however we are eager to swap the hard disk out with a replacement.

We are scheduling the disk change to occur tonight at 21:00

The downtime will be no longer than 30 minutes.

The affected node is: mysql360.cp.blacknight.com

We will update this post once completed.

UPDATE 21:17: This has now been completed.

We are currently experiencing issues with the following two mysql nodes:

mysql469.cp.blacknight.com

mysql496.cp.blacknight.com

Our engineers are working on resolving this asap and will update this blog post with more information when possible.

Update: 10:39: This issue has now been resolved.

We have been alerted to a disk failure in both PEMVZMPS20 and PEMVZMPS23. Due to it's RAID configuration no data loss has occured of course.

To get the RAID back up and running to it's full potential we are going to replace the failed disks tonight.

When: 3rd of March 2010 at 21:00 (Expected downtime 15 minutes per node.)

Affected nodes are:

81.17.254.85    pemlinweb04.blacknight.com     
81.17.254.86    pemlinweb05.blacknight.com
81.17.254.45    mysql71.cp.blacknight.com

We will update this blog post once completed.


Update 21:32 - PEMVZMPS23 has been completed, both pemlinweb04/05 are back online. We are waiting on a disk check to finish on PEMVZMPS20, once complete the node will be brought online. ETA 15 minutes.


Update: 22:20 - both nodes are back up and the services are restored. As the raid arrays are rebuilding thing sill be a little sluggish until it's completed. This will take several hours.

Due to increased traffic we are seeing on one of our MySQL nodes we are going to migrate it to it's own dedicated hardware.

The downtime will occur tonight at 22:00, Feb 8th 2010

The downtime for the affected node will be no more than 45 minutes.

The affected node is:

mysql106.cp.blacknight.com
mysql106int.cp.blacknight.com

We will update this post once completed.


Update: This work has now been completed.

We experienced some issues with one of our mysql nodes at 19:35. Some users may have seen connection timeout errors on their database connections for a peroid of 3/5 minutes.

The affected node was: mysql71.cp.blacknight.com

This has been resolved now and normal service is restored.