Recently in Database Category

We have been alerted to a disk failure in both PEMVZMPS20 and PEMVZMPS23. Due to it's RAID configuration no data loss has occured of course.

To get the RAID back up and running to it's full potential we are going to replace the failed disks tonight.

When: 3rd of March 2010 at 21:00 (Expected downtime 15 minutes per node.)

Affected nodes are:

81.17.254.85    pemlinweb04.blacknight.com     
81.17.254.86    pemlinweb05.blacknight.com
81.17.254.45    mysql71.cp.blacknight.com

We will update this blog post once completed.


Update 21:32 - PEMVZMPS23 has been completed, both pemlinweb04/05 are back online. We are waiting on a disk check to finish on PEMVZMPS20, once complete the node will be brought online. ETA 15 minutes.


Update: 22:20 - both nodes are back up and the services are restored. As the raid arrays are rebuilding thing sill be a little sluggish until it's completed. This will take several hours.

Due to increased traffic we are seeing on one of our MySQL nodes we are going to migrate it to it's own dedicated hardware.

The downtime will occur tonight at 22:00, Feb 8th 2010

The downtime for the affected node will be no more than 45 minutes.

The affected node is:

mysql106.cp.blacknight.com
mysql106int.cp.blacknight.com

We will update this post once completed.


Update: This work has now been completed.

We experienced some issues with one of our mysql nodes at 19:35. Some users may have seen connection timeout errors on their database connections for a peroid of 3/5 minutes.

The affected node was: mysql71.cp.blacknight.com

This has been resolved now and normal service is restored.

Due to a hardware upgrade we are migrating the MySQL node listed below to new hardware this evening. All details are listed below

When?
The downtime will occur tonight at 22:00 hours, 27th of January 2010

What will be offline?
Any databases located on these servers:
mysql360.cp.blacknight.com
mysql360int.cp.blacknight.com

For how long?
The downtime will last an hour.

We will update this post once completed.

Update 23:00

This update completed at 22:24. This mysql server node is now live on a new hw node and is already showing better performance for customers using it.