Blacknight Hosting Network Status

backup.mybackup.ie file system errors

Thu, 11 Mar 2010 08:06:30 +0000

Summary: One of the SAN devices that is connected to this cluster has a dodgy file system. It's currently being recovered but this will take upto 8 hours. There will be no data loss, this is semi expected as the file system hasn't been checked for over a year and it's recently had huge volumes of data removed from it.

Update: 11:00 am

This file system has been restored and the services are now back online.

Nominet Scheduled Maintenance April 11 2010

Tue, 09 Mar 2010 17:41:52 +0000

Nominet, who operate the co.uk registry, have given us advanced warning of a major maintenance window on Sunday April 11th 2010.

Most Nominet systems will be unavailable from 0700 to 1300, so we will be unable to process any new registrations or updates during this time.

The official announcement provides some more technical detail:

During this time Online Services including web domain manager and PRSS,
EPP, and ENUM registration systems will be unavailable. The automaton will
not process email requests, but they will be received and queued for
processing when it restarts. All of our registration systems will restart
simultaneously at 13:00, at which point EPP requests will be accepted and
the automaton will start to process queued requests - it is therefore
possible that an EPP request may be processed sooner than an earlier
automaton request for the same domain name.

WHOIS and DAC services will be moved to our backup data centre so will
continue to operate as normal. This upgrade is to our database software
only and no functionality changes will be made to our systems during this
maintenance outage.

Essential Maintenance PEMVZMPS20, PEMVZMPS23

Wed, 03 Mar 2010 13:24:39 +0000

We have been alerted to a disk failure in both PEMVZMPS20 and PEMVZMPS23. Due to it's RAID configuration no data loss has occured of course.

To get the RAID back up and running to it's full potential we are going to replace the failed disks tonight.

When: 3rd of March 2010 at 21:00 (Expected downtime 15 minutes per node.)

Affected nodes are:

81.17.254.85    pemlinweb04.blacknight.com
81.17.254.86    pemlinweb05.blacknight.com
81.17.254.45    mysql71.cp.blacknight.com

We will update this blog post once completed.

Update 21:32 - PEMVZMPS23 has been completed, both pemlinweb04/05 are back online. We are waiting on a disk check to finish on PEMVZMPS20, once complete the node will be brought online. ETA 15 minutes.

Update: 22:20 - both nodes are back up and the services are restored. As the raid arrays are rebuilding thing sill be a little sluggish until it's completed. This will take several hours.

Telnic - Telhosting Upgrade

Tue, 02 Mar 2010 09:20:24 +0000

Telnic are upgrading the Community Telhosting Platform (CTH) on Thursday 4th March 2010.

The upgrade will take place at 0600 UTC.

Production System Impact Summary

* CTH Web based UI and API - Available during the upgrade window though some session loss may be experienced
* SRS - Not affected and fully available
* DNS - Not affected and fully available
* WHOIS - Not affected and fully available
* TelProxy - Not affected and fully available

cp.blacknight.com scheduled downtime

Mon, 01 Mar 2010 09:25:19 +0000

Summary: Tonight Monday March 1st we're moving the Provisioning node known as OSSCORE to new hardware. This is due to the sheer volume of customers going into the system on a daily basis. This has caused our provisioning back end to get slower and slower which is causing cp.blacknight.com (web hosting) to slow down.

When: March 1st 2010 starting at 22:00 until 01:00 on March 2nd.

What: From 22:00 hours we'll be working on the migration, the cp on cp.blacknight.com will be turned off for the duration of this migration in order to prevent inconsistencies

Services affected: Management of your web hosting plans, e-mail, databases etc will not be available during this window but your hosting will be unaffected.

Update 23:00: this window has been completed successfully a full 2 hours early.

PEMVZWIN06 - Reboot

Thu, 25 Feb 2010 20:41:49 +0000

An emergency reboot will be taking place on PEMVZWIN06 at 02:00 on the 26th of February.

This will effect the following VPSs:

78.153.208.255
78.153.208.163
78.153.210.20
78.153.210.160
78.153.210.172
78.153.209.204
78.153.210.184
78.153.209.166
78.153.210.203
78.153.210.81

We apologise for any inconvenience caused

UPDATE 02:20: This has been completed without issue. All affected VPS' are back up.

pemwinweb08 having issues

Wed, 24 Feb 2010 08:16:10 +0000

Summary: pemwinweb08 is currently experiencing issues. Our engineering team are on the case and hopefully it'll return to normal service shortly. We'll post further updates here.

Update: 08:20 The server is back now. The Frontnet (public facing network adapter) disappeared. We believe this to be related to a bug in our provisioning software or some sort of configuration conflict. On reboot the interface came backup and the machine is back on the air. We're investigating this issue further.

cp.blacknight.com outage this morning Feb 22nd

Mon, 22 Feb 2010 09:07:48 +0000

Summary: Some of the services behind cp.blacknight.com are currently unstable which is causing the control panel to show 404 and proxy errors. We're working with our software vendor to resolve this as soon as possible. We hope to put the problem behind us this week once and for all. Further updates will be posted here when we have them.

Update: 09:38

cp.blacknight.com is now backup, but it's considered unstable until further notice. As mentioned above our vendor (Parallels) have been working on this for approx 2 weeks and twice they've informed us that it's been fixed, yet we keep seeing more problems. Further updates will be posted here when have them.

Store Issues

Sat, 20 Feb 2010 23:26:10 +0000

We've been experiencing some issues with the store this evening.

Our technical team are working on resolving the issue as quickly as possible

This should not have any impact on any existing services - just the actual shopfront.

UDPATE 10:00 - this issue has been resolved.

Maintenence Window - pemwinweb08 (81.17.250.65)

Wed, 17 Feb 2010 11:02:48 +0000

We wish to notify you of this 2 hour maintenence window on pemwinweb08 (81.17.250.65) shared Windows server, which will begin at 22:00 GMT tonight. All services hosted on this server may be unavailable for a period during this window.

We apologise in advance for any inconvenience caused.

cp.blacknight.com maintenance work

Thu, 11 Feb 2010 11:33:32 +0000

When: 03:00 to 06:00 on Friday Feb 12th.

Summary: In order to diagnose issues with the java service that drives the UI in our control panel our Software Vendor wishes to perform several restarts of the service. We'll minimise the hit on customers by doing this between 03:00 and 06:00 tomorrow Feb 12th.

What: In order to diagnose the problems which cause 404 tomcat errors and proxy gateway errors on cp.blacknight.com parallels want to put their application into "deep investigation" mode. This will involve several restarts and potentially prolonged bouts of downtime (10-15 minutes) during this maintenance window.

We'll hopefully see a more stable control panel once they've done this and found the root cause of the problem.

backup.mybackup.ie - Offline

Thu, 11 Feb 2010 08:26:36 +0000

The backup service, backup.mybackup.ie will be offline for the majority of the day due to some file system errors we've seen on one of it's SAN volumes.

Our engineers are working on this currently and this blog post will be updated once completed.

Update: 16:50

This file system check completed at approx 14:30. We restarted the system and it's now back working 100%. We've taken some steps to get notified quicker also if a file system changes from RW to RO (where the FS is damaged for some reason).

This issue is now resolved fully.

Ragnell Issues

Wed, 10 Feb 2010 15:16:35 +0000

Ragnell is currently having issues which is causing it to become unresponsive. There is an engineer on site getting it backup, and it should reappear within the next 10mins.

Hardware node issues pemvzmps32

Wed, 10 Feb 2010 15:04:51 +0000

Summary: The hardware node pemvzmps32 which contains pemlinweb31 and mysql452 is experiencing issues. This is resulting in strange behaviour such as ftp timing out and high load on the mysql node. We plan to move these two containers off of this node tonight between 22:00 and 01:00 at which time the issues that people are experiencing will be fixed. Unfortunately this can't be resolved any sooner without significant downtime.

Emergency Reboot - PEMVZWIN06

Tue, 09 Feb 2010 17:48:27 +0000

VZWIN06 Hardware node needs to be rebooted at 8:45 tonight

This will effect the following VPSs:

78.153.208.255
78.153.208.163
78.153.210.20
78.153.210.160
78.153.210.172
78.153.209.204
78.153.210.184
78.153.209.166
78.153.210.203
78.153.210.81

We apologise for any inconvenience caused