December 2010 Archives

We are currently experiencing licensing issues on PEMVZWIN14 which has caused it to shut down VPS running on it. We are working with the vendors to get it back up and running as soon as possible.

UPDATE 12:45: This has now been resolved and all VPS should be back up and running.

The above server has stopped responding. We have dispatched an engineer to have a look at it.

Update 11:30: The server was rebooted and all services & VPSs have returned to normal.

Summary: At approx 13:46 today we experienced some packet loss on our INEX facing connections. This resulted in some BGP flaps so our routes were withdrawn from several ISPs and then re-advertised. This would have appeared as a brief network outage. We've received a report from INEX explaining what caused this issue.

The report follows:

"Due to a port configuration issue, some INEX members may have experienced
packet loss today between 13:46:04 UTC and 13:50:46 UTC today (2010/12/23).

The reason for this packet loss was that the configuration on some
member-facing ports was changed to include unicast storm control.  While
this should have been a non service-affecting operation, it appears that
this was not the case.  We will shortly be opening up a support case about
this, as the packet loss was unexpected."

We don't expect any further issues as a result of this.

We currently experiencing issues with our shared linux server gorlois.blacknight.ie - Our engineers are working on this issue at the moment and will update this post once complete.

This server is currently experiencing a high load, and as such may be slow.

We are trying to bring the load down and restore normal services.

 

Update: We had to reboot this server, and it is now carrying out a file system check. We aplogise for this unexpected downtime.

Update 2: The file system check has been completed and all services have returned to normal.

Summary: As many of our customers will have noticed there have been problems with e-mail over the past 2 months or so. Our plan for this is simple. We've built a new storage system designed specifically to take the current load. The move to this new platform will be in two parts. Firstly consolidate all mail equipment to a rack with enough space for expansion. The second phase which we'll announce next week after the successful move of the servers.

The first part of the plan is to move all the existing to a new rack in the same Cage. On Monday 20th of December at 07:00 we will shutdown e-mail and proceed to re-rack the equipment.

This will mean there will be a 1 hour outage of e-mail services where you won't be able to check your e-mail or send any e-mail.

What's affected: All e-mail services for hosting accounts administered from cp.blacknight.com will be affected for a duration of 1 hour on the morning of Monday 20th of December.

Summary: At 10:26 on Thursday December 16th the above named server blue screened and has rebooted itself. Currently unsure if patch Tuesday is to blame. Our own internal SUS shouldn't have pushed updates to it, but it's possible they did.

The containers on this node are starting already and should be back by 10:50 approx. We'll post further updates on this thread if required.

Before completing this status update, all the containers are already fully back up and running. Please check to ensure that your own internal services have resumed correctly.

As Christmas is next week here are our opening hours over the break:

Friday 24th December 2010 - 0900 - 1300
Saturday 25th / Sunday 26th December - closed.
Monday 27th / Tuesday 28th December 1200 - 1600
Wednesday 29th, Thursday 30th, Friday 31st December - 0900 to 1800
Saturday 1st January 2011 and Sunday 2nd January 2011 - closed
Monday 3rd January 2011 1200 - 1600
Tuesday 4th January 2011 - normal office hours resume

NB: We have staff on duty 24/7/365. If you have a dedicated server or colo please refer to the "out of hours" contact details

And, in case we forget, have a good break!

We were forced to remotely reboot igraine as the load went up so high that it became unresponsive. Unfortunately on reboot, anfile system check was required that slowed things down. This has completed, and it should be back now.

We are currently experiencing issues with our hardware node PEMVZWIN04.

We are currently expierencing issues with the VPSs located on the hardware node: PEMVZLIN18.

Due to an error we were seeing on this linux shared hosting hardware node, a reboot was needed.

Eurid, the .eu registry, will be carrying out maintenance on their systems on Wednesday 15 December 2010 from 2100 to 2230 CET.

Registrations and updates will not be available during this period.

We are currently experiencing issues with email over mail.blacknight.com - Our engineers are working to resolve this issue currently. Updates will be posted here.

Due to the hardware node PEMVZMPS43 becoming unresponsive the server has been sent for a reboot.

We are currently experiencing issues with our Mail Clusters storage. Our engineers are working on diagnosing and resolving the issue asap.

Sometime before midnight last night, morgana was compromised with a root level exploit. It looks like any index file has been overwritten on all sites. Mail seems to be unaffected.

The most recent uncompromised backup seems to be 16:00 yesterday, so we're working to restore the overwritten html files from that.

When we've more data available, it will be put up here.

Update 15:30: Mail is back up and running, however we're running into delays restoring compromised index pages from the backups. The backups are fine, however the program for restoring them seems to be sulking due the presence of dead symlinks in the backup.

We are working with the vendors to resolve this and get a full restore. In the meantime, we're going to use a previously restored backup from Oct18th to get sites up and running again. Once the bug with more current restore is resolved, we will bring back up yesterday's versions of the files.

Update 01:30: The mass restore of files from Oct 18th is going at full swing and should be completed within an hour or two.

Update: 08:30 Monday Dec 6th.

The mass restore is taking a little longer than anticipated. it's working it's way through alphabetically currently. This will take probably another 16-24 hours but we might stop it and run it in a threaded fashion instead of a single thread which will be much faster.

There have been a large number of sites restored to their state from the 4th of December. However as index* in terms of filenames were overwritten there's many sub directories of websites not working.

Please open a support ticket and our support team can restore individual files for you. However we will ask for some patience as we want the mass restore to finish before we start doing further restores.

We are still working with the vendor in order to get the restore of yesterday's backups running properly. We do have a full backup, and we can easily restore any individual files or directories if there's something badly needed, however a mass restore like we can do on the backup from Oct 18th is failing.

December 6th @ 12:00

The restore of the full backup from October 18th 2010 has completed. The full CDP snapshot from December 4th @ 15:21 is at 65.7 GB of 111.1 GB. This will take another 2hours or so to restore at which point we're going to completely restore all public_html and private_html directories completely to this date/time.

If you have critical sites which are still down please contact us immediately via our helpdesk, however until this full restore is complete we can't access this snapshot so we can't do individual file restores.

December 6th @ 14:30

The restore from the back of Dec 4th @ 15:21 has now completed. This has been restored to another server. We are now proceeding with the restoration of changed files using rsync. This will take several hours to complete and we will update this post again once we're done.

We are now in a position to restore individual files on request from people so please let us know if there is something urgent that you need done.

December 6th @ 16:45

The restore of files from December 4th should now be complete. This issue is considered closed. If you're having any problems please let us know and we can restore files manually for you very quickly.