Summary: One of the SAN devices that is connected to this cluster has a dodgy file system. It's currently being recovered but this will take upto 8 hours. There will be no data loss, this is semi expected as the file system hasn't been checked for over a year and it's recently had huge volumes of data removed from it.

Update: 11:00 am

This file system has been restored and the services are now back online.

Nominet, who operate the co.uk registry, have given us advanced warning of a major maintenance window on Sunday April 11th 2010.

Most Nominet systems will be unavailable from 0700 to 1300, so we will be unable to process any new registrations or updates during this time.

The official announcement provides some more technical detail:

During this time Online Services including web domain manager and PRSS,
EPP, and ENUM registration systems will be unavailable. The automaton will
not process email requests, but they will be received and queued for
processing when it restarts. All of our registration systems will restart
simultaneously at 13:00, at which point EPP requests will be accepted and
the automaton will start to process queued requests - it is therefore
possible that an EPP request may be processed sooner than an earlier
automaton request for the same domain name.

WHOIS and DAC services will be moved to our backup data centre so will
continue to operate as normal. This upgrade is to our database software
only and no functionality changes will be made to our systems during this
maintenance outage.

We have been alerted to a disk failure in both PEMVZMPS20 and PEMVZMPS23. Due to it's RAID configuration no data loss has occured of course.

To get the RAID back up and running to it's full potential we are going to replace the failed disks tonight.

When: 3rd of March 2010 at 21:00 (Expected downtime 15 minutes per node.)

Affected nodes are:

81.17.254.85    pemlinweb04.blacknight.com     
81.17.254.86    pemlinweb05.blacknight.com
81.17.254.45    mysql71.cp.blacknight.com

We will update this blog post once completed.


Update 21:32 - PEMVZMPS23 has been completed, both pemlinweb04/05 are back online. We are waiting on a disk check to finish on PEMVZMPS20, once complete the node will be brought online. ETA 15 minutes.


Update: 22:20 - both nodes are back up and the services are restored. As the raid arrays are rebuilding thing sill be a little sluggish until it's completed. This will take several hours.

Telnic are upgrading the Community Telhosting Platform (CTH) on Thursday 4th March 2010.

The upgrade will take place at 0600 UTC.

Production System Impact Summary

* CTH Web based UI and API - Available during the upgrade window though some session loss may be experienced
* SRS - Not affected and fully available
* DNS - Not affected and fully available
* WHOIS - Not affected and fully available
* TelProxy - Not affected and fully available

Summary: Tonight Monday March 1st we're moving the Provisioning node known as OSSCORE to new hardware. This is due to the sheer volume of customers going into the system on a daily basis. This has caused our provisioning back end to get slower and slower which is causing cp.blacknight.com (web hosting) to slow down.

When: March 1st 2010 starting at 22:00 until 01:00 on March 2nd.

What: From 22:00 hours we'll be working on the migration, the cp on cp.blacknight.com will be turned off for the duration of this migration in order to prevent inconsistencies

Services affected: Management of your web hosting plans, e-mail, databases etc will not be available during this window but your hosting will be unaffected.

Update 23:00: this window has been completed successfully a full 2 hours early.

An emergency reboot will be taking place on PEMVZWIN06 at 02:00 on the 26th of February.

This will effect the following VPSs:

78.153.208.255
78.153.208.163
78.153.210.20
78.153.210.160
78.153.210.172
78.153.209.204
78.153.210.184
78.153.209.166
78.153.210.203
78.153.210.81

We apologise for any inconvenience caused

UPDATE 02:20: This has been completed without issue. All affected VPS' are back up.

Summary: pemwinweb08 is currently experiencing issues. Our engineering team are on the case and hopefully it'll return to normal service shortly. We'll post further updates here.

Update: 08:20 The server is back now. The Frontnet (public facing network adapter) disappeared. We believe this to be related to a bug in our provisioning software or some sort of configuration conflict. On reboot the interface came backup and the machine is back on the air. We're investigating this issue further.

Summary: Some of the services behind cp.blacknight.com are currently unstable which is causing the control panel to show 404 and proxy errors. We're working with our software vendor to resolve this as soon as possible. We hope to put the problem behind us this week once and for all. Further updates will be posted here when we have them.

Update: 09:38

cp.blacknight.com is now backup, but it's considered unstable until further notice. As mentioned above our vendor (Parallels) have been working on this for approx 2 weeks and twice they've informed us that it's been fixed, yet we keep seeing more problems. Further updates will be posted here when have them.

We've been experiencing some issues with the store this evening.

Our technical team are working on resolving the issue as quickly as possible

This should not have any impact on any existing services - just the actual shopfront.

UDPATE 10:00 - this issue has been resolved.

We wish to notify you of this 2 hour maintenence window on pemwinweb08 (81.17.250.65) shared Windows server, which will begin at 22:00 GMT tonight. All services hosted on this server may be unavailable for a period during this window.

 

We apologise in advance for any inconvenience caused.