March 2010 Archives

We are currently experiencing issues with the shared hosting linux servers ector and igraine.

Our engineers are working on this issue at present and will update this post as soon as it is resolved.

UPDATE 14:47 - These issues have now been resolved.

UPDATE 15:14 - This issue has presented itself again on ector, we are currently resolving.

We are currently experiencing issues with our shared windows hosting server: palamedes

Out engineers are working their way to the data center currently to resolve. ETA 30 minutes.

UPDATE 02:21: This issue has been resolved.

We are currently experiencing issues with the shared linux server iseult.

Our engineers are working on this currently and we will update you once resolved.

UPDATE: This has now been resolved.

A hard disk has failed on our shared hosting hardware node PEMVZMPS35.

No data loss has occurred due to our RAID configuration on the server however we are eager to swap the hard disk out with a replacement.

We are scheduling the disk change to occur tonight at 21:00

The downtime will be no longer than 30 minutes.

The affected node is: mysql360.cp.blacknight.com

We will update this post once completed.

UPDATE 21:17: This has now been completed.

Gorlois has become unresponsive. There's currently an engineer on the way to see what the problem is and bring it back up.

We have been informed that the .com registry will be undergoing maintenance on Saturday March 27th 2010 from 0100 to 0145 UTC

Existing .com domains will not be affected, but updates and new registrations will not be processed during this time

We will be scheduling a reboot of the Windows Shared Hosting server: pemwinweb02 - 81.17.250.70.

The downtime for the node will be no more than 10 minutes.

The downtime will be occurring at 21:00 tonight the 19th of March.

Thank you for you cooperation

We will of course update this post once completed.

Update: This work has been complete.

The above server is experiencing problems at the moment. Our engineers are aware of the issue and will provide an update soon.

 

Update 11:06 - This problem has been resolved.

We are currently experiencing some issues with our control panel http://cp.blacknight.com

Our engineers are working on this currently and will update this post once resolved.

Update 10:01: This was resolved at 09:43 apologies for the delay in updating the ticket.

We are currently experiencing issues with two of the MySQL nodes in our shared cluster

Our technical team are aware of the issue and are working on resolving it as quickly as possible

The nodes affected are:
mysql469.cp.blacknight.com

mysql496.cp.blacknight.com

UPDATE: 2100 - this issue appears to have been resolved, however our technical team will be keeping a close eye on the hardware node in question

We are currently experiencing issues with our linux shared hosting node: balin

Our engineers are working on resolving this asap. This blog post will be updated once resolved.


Update: 16:47: This issue has now been resolved.

We are currently experiencing issues with the following two mysql nodes:

mysql469.cp.blacknight.com

mysql496.cp.blacknight.com

Our engineers are working on resolving this asap and will update this blog post with more information when possible.

Update: 10:39: This issue has now been resolved.

Our main site is currently not accessible due to a network issue

The store, control panel, support desk and all services related to hosting, domains, dns etc., are NOT affected

We are currently working on resolving the issue as quickly as possible

UPDATE 2208 - issue has been resolved

Since tomorrow is St Patrick's Day, which is a national holiday in Ireland, our offices are open, but for a reduced period

Our offices will be open between 1200 and 1600

Dedicated / colo clients - please use the out of hours emergency number outside those hours.

We are currently experiencing an issue with the control panel which is preventing users from logging in.

Our technical team is aware of the situation and is working on it.

UPDATE 1433 - the control panel is currently accessible and we will update this post if we get any more details or see any further issues

Summary: One of the SAN devices that is connected to this cluster has a dodgy file system. It's currently being recovered but this will take upto 8 hours. There will be no data loss, this is semi expected as the file system hasn't been checked for over a year and it's recently had huge volumes of data removed from it.

Update: 11:00 am

This file system has been restored and the services are now back online.

Nominet, who operate the co.uk registry, have given us advanced warning of a major maintenance window on Sunday April 11th 2010.

Most Nominet systems will be unavailable from 0700 to 1300, so we will be unable to process any new registrations or updates during this time.

The official announcement provides some more technical detail:

During this time Online Services including web domain manager and PRSS,
EPP, and ENUM registration systems will be unavailable. The automaton will
not process email requests, but they will be received and queued for
processing when it restarts. All of our registration systems will restart
simultaneously at 13:00, at which point EPP requests will be accepted and
the automaton will start to process queued requests - it is therefore
possible that an EPP request may be processed sooner than an earlier
automaton request for the same domain name.

WHOIS and DAC services will be moved to our backup data centre so will
continue to operate as normal. This upgrade is to our database software
only and no functionality changes will be made to our systems during this
maintenance outage.

We have been alerted to a disk failure in both PEMVZMPS20 and PEMVZMPS23. Due to it's RAID configuration no data loss has occured of course.

To get the RAID back up and running to it's full potential we are going to replace the failed disks tonight.

When: 3rd of March 2010 at 21:00 (Expected downtime 15 minutes per node.)

Affected nodes are:

81.17.254.85    pemlinweb04.blacknight.com     
81.17.254.86    pemlinweb05.blacknight.com
81.17.254.45    mysql71.cp.blacknight.com

We will update this blog post once completed.


Update 21:32 - PEMVZMPS23 has been completed, both pemlinweb04/05 are back online. We are waiting on a disk check to finish on PEMVZMPS20, once complete the node will be brought online. ETA 15 minutes.


Update: 22:20 - both nodes are back up and the services are restored. As the raid arrays are rebuilding thing sill be a little sluggish until it's completed. This will take several hours.

Telnic are upgrading the Community Telhosting Platform (CTH) on Thursday 4th March 2010.

The upgrade will take place at 0600 UTC.

Production System Impact Summary

* CTH Web based UI and API - Available during the upgrade window though some session loss may be experienced
* SRS - Not affected and fully available
* DNS - Not affected and fully available
* WHOIS - Not affected and fully available
* TelProxy - Not affected and fully available

Summary: Tonight Monday March 1st we're moving the Provisioning node known as OSSCORE to new hardware. This is due to the sheer volume of customers going into the system on a daily basis. This has caused our provisioning back end to get slower and slower which is causing cp.blacknight.com (web hosting) to slow down.

When: March 1st 2010 starting at 22:00 until 01:00 on March 2nd.

What: From 22:00 hours we'll be working on the migration, the cp on cp.blacknight.com will be turned off for the duration of this migration in order to prevent inconsistencies

Services affected: Management of your web hosting plans, e-mail, databases etc will not be available during this window but your hosting will be unaffected.

Update 23:00: this window has been completed successfully a full 2 hours early.