March 2011 Archives

Summary: This evening around 21:31 GMT all websites on pemlinweb10 were defaced. In most cases additional files called index.htm, index.html and index.php were put into document roots for websites.

The action we've taken thus far:

1) disabled all the files
2) removed all the files (currently underway still) so that you can upload your site files if you have backups.
3) kicked off a bare metal restore of the servers file system to one of our dedicated restore servers.
4) found the entry point, hole and have plugged them.

What's left: The bare metal restore will take approx 9 hours to complete, it's currently one hour into this so it'll be finished by morning when our engineering shift starts at 8am. We will then begin a mass restore of sites from the restore. The latest backup for this server is started at 15:00 this evening and ended at 18:31 so the data should be quite recent. And infact since it was just index files it shouldn't take a lot of effort to restore them once the full restore has taken place.

Caveats: Anyone who uploads a new index files this evening should expect this to be overwritten from our restore tomorrow morning. If the restore fails for some reason we'll have to kick it off again. However we have other options available to us but a bare metal restore is typically the fastest method.

Update 08:31: The restore is still on going, the ETA is around 3h and 30 minutes which brings us upto midday or there abouts. We'll post further updates when we have the restore completed.

FYI we just rebooted the hardware node that pemlinweb10 resides on and it should be back in a few minutes.

Update 12:10: At the moment any file who's name starts with index is being restored to the state it was in at 15:00 yesterday. We don't have an ETA on how long this part will take, but it should hopefully be finished before lunch.

Update 16:30: The restore has taken longer than expected, mainly because we vastly under estimated just how many "index" files there are. For example, Joomla puts an index.html in every folder as a measure against directory listings. The vast majority of files are  restored now, with only a few thousand left. The process should be finished within the next 15 mins.

I'm afraid that two of our Linux shared hosting servers are currently experiencing technical issues due to abnormally high load.  They are:

Pemlinweb21 - 81.17.254.58
Pemlinweb22 - 81.17.254.59

Our engineers are working on stopping the processes using up all the resources on the servers now and we hope to have service restored as soon as possible.

Update 14:56 - The node itself has been restarted now and this has restored service fully.  Our engineers will continue to monitor the servers closely, but what we believe to be the primary cause of this has been stopped fully now.

We are currently experiencing a larger than average number of emails in the mailqueues of the shared mailservers for any shared hosting package using email through cp.blacknight.com.

This may cause a delay in your emails getting delivered to you.  Our engineers are working on this and it is their highest priority right now.  We hope to get the situation resolved as quickly as possible.


Update 10:54 - I'm afraid the mail queues are still quite high, though our engineers have blocked one possible contributer to the high backlog.  We have escalated this to the software developers also.

Update 12:40 - Apologies over the delay in an update.  The mailqueues started to go down quickly shortly after the change made at 10:54 but our engineers could not see the exact cause.  They are still closely monitoring the mailservers and working with the Software Developers but email delivery has been down to normal levels for the last half or so now.

We are currently seeing some errors coming from virtuozzo on PEMVZLIN08. In order to patch these we are going to need to reboot the node. The downtime will be in the region on 20-25 minutes and the following VEs will be affected:

Next Wednesday we will be moving our Shared DirectAdmin and Helm hosts behind a dedicated pair of firewalls. This will involve a certain amount of inevitable downtime, however we will try and keep it to a minimum. Total downtime should be less than 30mins.

As part of this, we'll also be doing a reboot of all our DirectAdmin machines to ensure that they're on the latest kernel.

If you log into your hosting via http://cp.blacknight.com you will not be affected.

UPDATE 23/03/2011 23:35: This work has been completed and all services seem to be back up and running.

Summary: Pemlinweb23 and 24 are currently experiencing some issues. We're working on the issue at the moment and we hope to have them back shortly.

Services affected: Websites hosted on pemlinweb23 and pemlinweb24 including ftp services.

Update 23:45: Both these servers are back working 100%

Igraine has stopped responding. An engineer is currently on site getting it back up and running.

UPDATE 16:50: The server is back up and running and we're investigating why it went down in the first place. 

We are currently experiencing issues with a hardware node that houses the following:

We're just after having an serious network issue and things should be back up and running now.

The root cause was a DOS attempt from within our network which overloaded the firewalls.

We're currently making sure that all services are backup.

I'm afraid we are currently experiencing technical difficulties on the newer hosting package control panel system.  That is the control panel at:

http://cp.blacknight.com

Our developers are working on this currently and we hope to have service restored fully as soon as possible.


14:20 - Service should be fully restored now, though our developers are still closely monitoring the service just in case.

We are experiencing high volumes of emails on our mail cluster at the moment which is causing delays in mail delivery.

We are currently experiencing issues with our linux shared hosting node: PEMVZMPS19.

We are seeing memory leaks happening on a hardware node that serves our a couple of MySQL servers. Specifically:

Summary: Following on from http://blacknig.ht/29j we've noticed some file system issues with these two nodes. We're taking them down immediately and we're going to fix this now.

Domains on the following IPs are affected:

78.153.214.38
78.153.214.39

UPDATE 13:45: Both webservers are back up and the file systems are looking a lot healthier.

Update 14:20: We noticed that while the servers themselves started ok, apache didn't for some reason. We've rectified this now and we're working on a permanent solution. Anyone still experiencing problems please contact support ASAP.

PEMLINWEB47 and PEMLINWEB48 are currently unresponsive. We have an engineer on site investigating and we'll have them up as soon as possible again.

Summary: Two servers that are on pemvzmps39 (pemlinweb31 and mysql452) are having problems this morning. The server is going to have to be taken down and work performed on it.

Customers affected will be people using mysql452int.cp.blacknight.com (172.16.3.64) for their databases and pemlinweb31 for their webspaces will experience issues.

On pemlinweb31 the following IPs are affected:

81.17.254.38 81.17.255.45 81.17.255.46 81.17.254.193 81.17.255.100 81.17.255.10 - sites may work but may be unable to upload new files and ftp will be problematic.

We're working to resolve this now.

Due to a software error we are seeing with virtuozzo on the windows vps hardware node, PEMVZWIN02, we are going to need to reboot it ASAP. We're sorry for the downtime this causes but it's urgent enough to warrant a reboot it urgently.

Summary: A DOS (1) directed towards one of our BGP transit customers was impairing one of our routers. As this router carriers much or our traffic via Global Crossing traffic across it wasn't working as normal. The routes in question which were being attacked have been withdrawn from our network and it is now stable again.

Services affected: Cp, E-mail, Websites, VPS servers and Dedicated servers were all affected by this DOS attack

(1) Denial of Service attack http://en.wikipedia.org/wiki/Denial-of-service_attack