Blacknight Hosting Network Status

PEMWINWEB02 - Scheduled Reboot

2010-03-19T16:51:34Z

We will be scheduling a reboot of the Windows Shared Hosting server: pemwinweb02 - 81.17.250.70.

The downtime for the node will be no more than 10 minutes.

The downtime will be occurring at 21:00 tonight the 19th of March.

Thank you for you cooperation

We will of course update this post once completed.

Update: This work has been complete.

pemwinweb14 (81.17.250.47) experiencing problems

2010-03-19T11:02:44Z

The above server is experiencing problems at the moment. Our engineers are aware of the issue and will provide an update soon.

Update 11:06 - This problem has been resolved.

cp.blacknight.com Issues

2010-03-19T09:28:51Z

We are currently experiencing some issues with our control panel http://cp.blacknight.com

Our engineers are working on this currently and will update this post once resolved.

Update 10:01: This was resolved at 09:43 apologies for the delay in updating the ticket.

MySQL Node Issues

2010-03-18T20:23:48Z

We are currently experiencing issues with two of the MySQL nodes in our shared cluster

Our technical team are aware of the issue and are working on resolving it as quickly as possible

The nodes affected are:
mysql469.cp.blacknight.com

mysql496.cp.blacknight.com

UPDATE: 2100 - this issue appears to have been resolved, however our technical team will be keeping a close eye on the hardware node in question

Shared Hosting Server Balin Issues

2010-03-18T16:20:10Z

We are currently experiencing issues with our linux shared hosting node: balin

Our engineers are working on resolving this asap. This blog post will be updated once resolved.

Update: 16:47: This issue has now been resolved.

MYSQL Server Issues

2010-03-18T10:22:02Z

We are currently experiencing issues with the following two mysql nodes:

mysql469.cp.blacknight.com

mysql496.cp.blacknight.com

Our engineers are working on resolving this asap and will update this blog post with more information when possible.

Update: 10:39: This issue has now been resolved.

Main Site Offline

2010-03-16T21:56:02Z

Our main site is currently not accessible due to a network issue

The store, control panel, support desk and all services related to hosting, domains, dns etc., are NOT affected

We are currently working on resolving the issue as quickly as possible

UPDATE 2208 - issue has been resolved

St Patrick's Day Opening Hours

2010-03-16T20:24:51Z

Since tomorrow is St Patrick's Day, which is a national holiday in Ireland, our offices are open, but for a reduced period

Our offices will be open between 1200 and 1600

Dedicated / colo clients - please use the out of hours emergency number outside those hours.

Issues With CP.Blacknight.com

2010-03-13T13:50:19Z

We are currently experiencing an issue with the control panel which is preventing users from logging in.

Our technical team is aware of the situation and is working on it.

UPDATE 1433 - the control panel is currently accessible and we will update this post if we get any more details or see any further issues

backup.mybackup.ie file system errors

2010-03-11T08:06:30Z

Summary: One of the SAN devices that is connected to this cluster has a dodgy file system. It's currently being recovered but this will take upto 8 hours. There will be no data loss, this is semi expected as the file system hasn't been checked for over a year and it's recently had huge volumes of data removed from it.

Update: 11:00 am

This file system has been restored and the services are now back online.

Nominet Scheduled Maintenance April 11 2010

2010-03-09T17:41:52Z

Nominet, who operate the co.uk registry, have given us advanced warning of a major maintenance window on Sunday April 11th 2010.

Most Nominet systems will be unavailable from 0700 to 1300, so we will be unable to process any new registrations or updates during this time.

The official announcement provides some more technical detail:

During this time Online Services including web domain manager and PRSS,
EPP, and ENUM registration systems will be unavailable. The automaton will
not process email requests, but they will be received and queued for
processing when it restarts. All of our registration systems will restart
simultaneously at 13:00, at which point EPP requests will be accepted and
the automaton will start to process queued requests - it is therefore
possible that an EPP request may be processed sooner than an earlier
automaton request for the same domain name.

WHOIS and DAC services will be moved to our backup data centre so will
continue to operate as normal. This upgrade is to our database software
only and no functionality changes will be made to our systems during this
maintenance outage.

Essential Maintenance PEMVZMPS20, PEMVZMPS23

2010-03-03T13:24:39Z

We have been alerted to a disk failure in both PEMVZMPS20 and PEMVZMPS23. Due to it's RAID configuration no data loss has occured of course.

To get the RAID back up and running to it's full potential we are going to replace the failed disks tonight.

When: 3rd of March 2010 at 21:00 (Expected downtime 15 minutes per node.)

Affected nodes are:

81.17.254.85    pemlinweb04.blacknight.com
81.17.254.86    pemlinweb05.blacknight.com
81.17.254.45    mysql71.cp.blacknight.com

We will update this blog post once completed.

Update 21:32 - PEMVZMPS23 has been completed, both pemlinweb04/05 are back online. We are waiting on a disk check to finish on PEMVZMPS20, once complete the node will be brought online. ETA 15 minutes.

Update: 22:20 - both nodes are back up and the services are restored. As the raid arrays are rebuilding thing sill be a little sluggish until it's completed. This will take several hours.

Telnic - Telhosting Upgrade

2010-03-02T09:20:24Z

Telnic are upgrading the Community Telhosting Platform (CTH) on Thursday 4th March 2010.

The upgrade will take place at 0600 UTC.

Production System Impact Summary

* CTH Web based UI and API - Available during the upgrade window though some session loss may be experienced
* SRS - Not affected and fully available
* DNS - Not affected and fully available
* WHOIS - Not affected and fully available
* TelProxy - Not affected and fully available

cp.blacknight.com scheduled downtime

2010-03-01T09:25:19Z

Summary: Tonight Monday March 1st we're moving the Provisioning node known as OSSCORE to new hardware. This is due to the sheer volume of customers going into the system on a daily basis. This has caused our provisioning back end to get slower and slower which is causing cp.blacknight.com (web hosting) to slow down.

When: March 1st 2010 starting at 22:00 until 01:00 on March 2nd.

What: From 22:00 hours we'll be working on the migration, the cp on cp.blacknight.com will be turned off for the duration of this migration in order to prevent inconsistencies

Services affected: Management of your web hosting plans, e-mail, databases etc will not be available during this window but your hosting will be unaffected.

Update 23:00: this window has been completed successfully a full 2 hours early.

PEMVZWIN06 - Reboot

2010-02-25T20:41:49Z

An emergency reboot will be taking place on PEMVZWIN06 at 02:00 on the 26th of February.

This will effect the following VPSs:

78.153.208.255
78.153.208.163
78.153.210.20
78.153.210.160
78.153.210.172
78.153.209.204
78.153.210.184
78.153.209.166
78.153.210.203
78.153.210.81

We apologise for any inconvenience caused

UPDATE 02:20: This has been completed without issue. All affected VPS' are back up.