May 2010 Archives

At 23:00 tonight all Windows Shared Hosting servers accessed under cp.blacknight.com will be rebooted. Downtime will be minimal.

 

We are currently experiencing some issues on our newer shared mailservers and this will affect any customers that have email through a shared hosting package set up on http://cp.blacknight.com - though not our Hosted Exchange customers.

The issue is only with logging in to access your emails so any emails sent to your address during this time will be sent on to your mailbox and be ready for you to download as soon as this issue is resolved.

Currently our engineering are working on this problem and it is related to the service that handles the login and authentication of users accessing their email.

UPDATE 1138
Service is currently degraded, but functional. We are still working on the situation.

UPDATE 1155
Service has been restored, however we will be monitoring it closely

We have been notified that there will be scheduled maintenance on the .tel and .biz domain registries taking place on Saturday June 5th 2010

During this period new registrations and updates for .tel and .biz domain names will not be available.

The window runs from 1200 UTC to 1800 UTC

We are currently experiencing issues with our shared hosting server Ector.

Our engineers are working to resolve this ASAP.

Updates will be posted here as they become available.

UPDATE 4:31PM - This issue is now resolved.

As we work to resolve the domain registration issue we mentioned earlier we have taken the order system offline

UPDATE 1215

We have re-enabled hosting sales. Domain sales are currently not available

UPDATE 1230

All systems are back up and running again

We have been contacted by the IE Domain Registry in relation to DNS servers that are vulnerable to attack and exploit.

While the Blacknight DNS servers are NOT open to exploit using the "Kaminsky exploit" unfortunately a lot of other DNS servers are.

We will be attempting to contact as many of the affected registrants as possible to warn them of the issue.

If you want to test your DNS there is a free tool available here

en We are currently experiencing issues processing domain registrations and updates.

This is due to a software issue on our backend

Our development team are working on resolving this issue and we will let you know once we have an update

UPDATE 1230

This issue has now been resolved

The core access switches in Interxion are being upgraded to new hardware next Wednesday at 3 in the morning. There should be no disruption of service during this upgrade as all racks have multiple links back to the core of the network, however it is possible that there will be slight blips in connectivity as the switches figure out their new paths back to the core.

UPDATE: Unfortunately, due to unforeseen issues with cabling, we were unable to complete this upgrade. While attempting to sort out the offending cabling, some parts of the network were rendered inaccessible for a period of time.

We're going to reschedule this upgrade for some night next week.

We've noticed increased utilization on some VEs located on the hardware node PEMVZMPS14. As a result we would like to perform a memory upgrade on the server.

The downtime for the node will be minimal at aprox 10/15 minutes.

The affected nodes are MySQL nodes and are as follows:

mysql242.cp.blacknight.com
mysql245.cp.blacknight.com

The downtime will occur on 25th of May 2010 at 08:00AM
UPDATE: This has been rescheduled to the 26th of May 2010 at 8:00AM

Thank you for your understanding and we will update this blog post as soon as the work is completed.

UPDATE 08:07AM: This has now been completed.

Summary: pemlinweb31 and mysql452 are down due to disk failures in the above node. When the primary disk in the mirror failed it marked all the file systems as read only. As a result mysql452 went down right away as it was unable to write to the databases. Pemlinweb31's httpd when down around 4am after the nightly log rotation as part of this apache gets restarted.

An engineer has been deployed to our data centre to resolve this issue and it should be back within a couple of hours.

FYI if the machine needs to be rebuilt a bare metal restore from our backups will take 12+ hours but no dataloss should occur.

Update: 11:45

We've checked that the machine is in good working order i.e. there is no data loss. We're now replacing the dead disk and the server will be back online in approx 10 minutes.

Update: 12:00

The server is back up and running, mysql452 is back up and running. pemlinweb31 is currently doing a quota check during the boot process and it's taking a bit of time. It should be back shortly.

Update: 12:07

Both mysql452 and pemlinweb31 are fully back up and running. The old disk was replaced and the raid array is rebuilding currently.

Further updates will be posted here when we have them.

We have been alerted on some failed disks showing on both VPS hardware nodes pemvzlin06 and pemvzlin07.

The data is safe due to it's RAID configuration however we do like to nip these problems in the bud asap and have the RAID back up to 100%.

In order to do this, the disks will need replacing and the RAID will need to be rebuilt.

The downtime will occur tonight at 21:00 - 17/05/2010

The maintenance window will be between 21:00 and 22:00 - Your VPSs won't be offline for this long but we like to allow for contingencies.

The affected VPSs are:

78.153.208.243  vps-1003934-473.cp.blacknight.com
78.153.209.227  vps-1015517-767.cp.blacknight.com
78.153.208.209  vps-1021874-919.cp.blacknight.com
78.153.210.51   vps-1021985-920.cp.blacknight.com
78.153.210.55   vps-1022053-923.cp.blacknight.com
78.153.210.63   vps-1022204-927.cp.blacknight.com
78.153.210.66   vps-1022235-930.cp.blacknight.com
78.153.210.67   dtech.blacknight.com           
78.153.210.77   vps-1023314-976.cp.blacknight.com
78.153.210.65   vps-1023837-987.cp.blacknight.com
78.153.209.90   vps-1024586-991.cp.blacknight.com
78.153.210.94   vps-1024910-999.cp.blacknight.com
78.153.209.167  vps.surebet.ie                 
78.153.210.62   vps-1026068-1007.cp.blacknight.com
78.153.210.107  vps-1026881-1016.cp.blacknight.com
78.153.210.110  vps-1027048-1018.cp.blacknight.com
78.153.210.112  vps-1024706-1020.cp.blacknight.com
78.153.210.114  zenoss.blacknight.com          
78.153.210.124  vps-1027811-1032.cp.blacknight.com
78.153.210.134  vps-1028972-1049.cp.blacknight.com
78.153.208.166  vps-1029626-1053.cp.blacknight.com
78.153.210.83   vps-1029847-1061.cp.blacknight.com
78.153.210.142  vps-1030341-1067.cp.blacknight.com
78.153.210.143  vps-1030210-1068.cp.blacknight.com
78.153.210.135  vps-1030008-1070.cp.blacknight.com
78.153.208.174  vps-1031717-1089.cp.blacknight.com
78.153.210.230  vps-1041091-1222.cp.blacknight.com
78.153.210.236  vps-1041791-1227.cp.blacknight.com
78.153.209.18   vps-1000474-1249.cp.blacknight.com
78.153.210.252  vps-1048799-1254.cp.blacknight.com
78.153.209.157  eurocord-ed.org                
78.153.209.183  vps-1018761-855.cp.blacknight.com
78.153.209.184  vps-1018719-856.cp.blacknight.com
78.153.209.205  vps-1018861-858.cp.blacknight.com
78.153.209.193  vps-1018935-859.cp.blacknight.com
78.153.210.7    vps-1019223-870.cp.blacknight.com
78.153.210.9    vps-1019733-882.cp.blacknight.com
78.153.208.165  vps-1019343-884.cp.blacknight.com
78.153.210.24   vps-1019732-895.cp.blacknight.com
78.153.210.34   vps-1020519-902.cp.blacknight.com
78.153.209.244  vps-1020736-905.cp.blacknight.com
78.153.210.41   vps-1020954-909.cp.blacknight.com
78.153.210.84   vps-1023553-981.cp.blacknight.com
78.153.210.87   ironhide.jotel.net             
78.153.209.25   vps-1023867-989.cp.blacknight.com
78.153.210.82   vps-1024960-995.cp.blacknight.com
78.153.210.98   vps-1025606-1003.cp.blacknight.com
78.153.210.96   vps-1026744-1011.cp.blacknight.com
78.153.210.108  vps-1026892-1017.cp.blacknight.com
78.153.210.68   vps-1027125-1022.cp.blacknight.com
78.153.210.123  vps-1027797-1031.cp.blacknight.com
78.153.210.125  vps-1027807-1033.cp.blacknight.com
78.153.208.159  vps-1029637-1052.cp.blacknight.com
78.153.209.206  vps-1029953-1060.cp.blacknight.com
78.153.210.231  vps-1041208-1223.cp.blacknight.com
78.153.210.239  vps-1042062-1229.cp.blacknight.com
78.153.210.248  vps-1011696-1235.cp.blacknight.com

We will update this post with status updates as soon as they are available.


UPDATE 21:49 - This has been completed.

Summary: One of our virtuozzo nodes is currently experiencing problems. We're working on this now and hope to have it back in full working order in the next 30 minutes or so.

Update: 14:45 This service is now running as normal

Ector is currently experiencing file system issues which have necessitated a reboot in order to get it back up and running. The server is now back up and running and we are monitoring the file system in question.

We are currently experiencing an issue with the hardware node PEMVZWIN01.

Our engineers are working on this issue currently and will update this blog post once resolved.

UPDATE 9:20AM - This issue has now been resolved.

Summary: pemvzwin04 is having some issues at the moment and all of the containers hosted on it are currently inoperable.

ETA to Fix: Our onsite engineering team are currently assessing the situation and we'll post back here again shortly once we've got a better idea of the problem. Current ETA is approx 1 hour.

Update 16:33: This issue is now resolved.

As per information outlined in http://www.blacknightstatus.com/2010/04/pemlinweb29pemlinweb30-migration.html we will be migrating the VE PEMLINWEB30 tonight.

This will occur at 21:00 tonight, 10/05/2010

We will update this blog post once completed.

We are currently experiencing issues with our shared hosting server Gorlois.

We are currently experiencing issues with our VPS Hardware node PEMVZWIN02.

Our engineers are on route to resolve this issue as soon as possible.

The affected nodes are:
78.153.208.116  VPS-238
78.153.208.123  VPS-258
78.153.208.16   VPS-262
78.153.208.127  VPS-282
78.153.208.28   VPS-287
78.153.209.211  VPS-288
78.153.208.154  VPS-317
78.153.208.168  VPS-331
78.153.208.149  VPS-337
78.153.208.170  VPS-341
78.153.208.172  VPS-342
78.153.208.169  VPS
78.153.208.171  VPS-344
78.153.209.151  VPS-346
78.153.208.15   VPS-347
78.153.208.62   VPS-353
78.153.208.176  VPS-354
78.153.208.177  VPS-355
78.153.208.179  VPS-357
78.153.208.184  VPS-362
78.153.210.75   VPS-387
78.153.208.205  VPS-390
78.153.208.218  VPS-407
78.153.208.222  VPS-409
78.153.208.223  VPS-410
78.153.209.118  VPS-620
78.153.209.160  VPS-664
78.153.209.164  VPS-667
78.153.209.176  PARTYCENTRAL
78.153.209.107  VPS-710



UPDATE 06:23AM - The hardware node is back online and all VPSs are coming online now.

Ector has become unresponsive and we're currently trying to get it back up and running. As soon as it is back we'll update this post.

On the morning of Saturday the 15th we're going to be upgrading the core routers distribution routers in Interxion. While the downtime should be minimum, there is likely to blips in connectivity during the upgrade.

The new Cisco routers are already in place and hooked up into the network parallel with the current Junipers where possible. However in order to make them the primary distribution routers there is extra configuration and cabling required.

While the network should work around any work that's being done, there is likely to be short outages as routes converge. These outages are likely to affect servers in both DEG and Interxion.

The work will be done between 03:00 and 06:00 on Saturday May 15th.

UPDATE: This work has been completed.

The .me registry will be conducting scheduled maintenance on the following date and time:
5 June 2010 between 15:00 UTC - 19:00 UTC.

During this period existing domains will not be affected

No new registrations or updates will be possible

There is an issue currently with some webspaces on our newer shared hosting plans (any hosting plan where the control panel is http://cp.blacknight.com).  We have raised this with the control panel vendors and hope to have it resolved as soon as possible.

Basically any changes you make to any domains might not go through and stay out of sync (have a yellow icon beside them under All Domains).  This can include changes like:

# Adding a domain
# Changing or adding hosting to a domain
# Password protecting a folder
# Adding a subdomain

It currently does not affect making DNS changes or nameserver changes as this issue is related to the webspace a domain is using itself.


Update 4th May 14:35:
Although the control panel vendors are still looking into the cause of this all tasks that were stuck have been processed now and any websites affected by this this morning or over the weekend should be ok now.  This might still affect any further changes you make however and webspaces are still showing as out of sync.

Update 5th May 08:30:
This issue was resolved last night by the control panel vendors and there have been no re-occurances overnight.  The issue was caused by an automated process that stopped running.  The hosting control panel system was restarted fully to restart this process.

We are currently working on resolving an issue which is preventing users from logging in to cp.blacknight.com

UPDATE 1540
This issue has now been resolved.