Recently in Security Category

Next Wednesday we will be moving our Shared DirectAdmin and Helm hosts behind a dedicated pair of firewalls. This will involve a certain amount of inevitable downtime, however we will try and keep it to a minimum. Total downtime should be less than 30mins.

As part of this, we'll also be doing a reboot of all our DirectAdmin machines to ensure that they're on the latest kernel.

If you log into your hosting via http://cp.blacknight.com you will not be affected.

UPDATE 23/03/2011 23:35: This work has been completed and all services seem to be back up and running.

Sometime before midnight last night, morgana was compromised with a root level exploit. It looks like any index file has been overwritten on all sites. Mail seems to be unaffected.

The most recent uncompromised backup seems to be 16:00 yesterday, so we're working to restore the overwritten html files from that.

When we've more data available, it will be put up here.

Update 15:30: Mail is back up and running, however we're running into delays restoring compromised index pages from the backups. The backups are fine, however the program for restoring them seems to be sulking due the presence of dead symlinks in the backup.

We are working with the vendors to resolve this and get a full restore. In the meantime, we're going to use a previously restored backup from Oct18th to get sites up and running again. Once the bug with more current restore is resolved, we will bring back up yesterday's versions of the files.

Update 01:30: The mass restore of files from Oct 18th is going at full swing and should be completed within an hour or two.

Update: 08:30 Monday Dec 6th.

The mass restore is taking a little longer than anticipated. it's working it's way through alphabetically currently. This will take probably another 16-24 hours but we might stop it and run it in a threaded fashion instead of a single thread which will be much faster.

There have been a large number of sites restored to their state from the 4th of December. However as index* in terms of filenames were overwritten there's many sub directories of websites not working.

Please open a support ticket and our support team can restore individual files for you. However we will ask for some patience as we want the mass restore to finish before we start doing further restores.

We are still working with the vendor in order to get the restore of yesterday's backups running properly. We do have a full backup, and we can easily restore any individual files or directories if there's something badly needed, however a mass restore like we can do on the backup from Oct 18th is failing.

December 6th @ 12:00

The restore of the full backup from October 18th 2010 has completed. The full CDP snapshot from December 4th @ 15:21 is at 65.7 GB of 111.1 GB. This will take another 2hours or so to restore at which point we're going to completely restore all public_html and private_html directories completely to this date/time.

If you have critical sites which are still down please contact us immediately via our helpdesk, however until this full restore is complete we can't access this snapshot so we can't do individual file restores.

December 6th @ 14:30

The restore from the back of Dec 4th @ 15:21 has now completed. This has been restored to another server. We are now proceeding with the restoration of changed files using rsync. This will take several hours to complete and we will update this post again once we're done.

We are now in a position to restore individual files on request from people so please let us know if there is something urgent that you need done.

December 6th @ 16:45

The restore of files from December 4th should now be complete. This issue is considered closed. If you're having any problems please let us know and we can restore files manually for you very quickly.

Summary: In the last week to 10 days we've noticed the odd Magento site getting compromised. In each case Magento was extremely out of date and allowed the attacker to gain full access of the site and the database. Obviously if you're storing customer information in your Magento store this is a potentially huge problem.

We recommend that anyone not running the latest stable Magento install (manual installs) and the latest available install from our Appvault that you upgrade immediately.

Steps to do this:

1) take a backup of your files + database
2) check your backup to see if there were any odd looking files left behind
    e.g. htaces.php,logo.php etc
3) upload the relevant patch / files
4) ensure your site is working properly

We recommend that everyone who uses Magento does this as soon as possible.

One of our core philosophies has always been to run a "clean" network where feasible.

With that in mind we have recently started contacting clients whose websites appear to be compromised with malicious content.

We are using Google's Safe Browsing Alerts service to track malicious activity across our entire network (as members of RIPE we run our own network and are not relying on a 3rd party)

One of the best ways to avoid these kind of issues is to keep any software on your site (and your computers) up to date and change your passwords regularly.

Related articles

• Google Launches Phishing Alert Service (securityweek.com)
• Google Adds Phishing URL Alerts Into Network Tool (nytimes.com)
• Google To Warn Admins Of Malware Infestations (informationweek.com)
• Google rolls out phishing URL alerts for admins (go.theregister.com)

Our shared hosting server Ragnell has been compromised, and the majority of the index.php's have been replaced with a hacked version. We have disabled all copies of the compromised index files already.

We are at the moment making sure the hole used is fixed before re-enabling Apache. As part of this, PHP is being upgraded to PHP5.

We are also going to see about restoring the disabled index files, however this is going to take longer. The backup system we use is geared towards full system backups, so restoring individual files is likely to take a while. If you have an uptodate copy of your index file, it will probably be faster if you get it uploaded yourself. This can be done even while Apache is down.

Update 1430: The upgrade of php / Apache is almost complete. Once it's finished we will be able start restoring index files from backups.

Update 1515: Apache is back up and running.  We are currently restoring the index files from backup. This is going to take a long time.

UPDATE 1615: If your site's index file has been restored or if you've restored it yourself let us know if there are any issues.

UPDATE 16:52: As restoring individual index files is proving to be far too unwieldy, we are currently restoring the whole partition to another box. This will allow us to script the restore of any index files which are still showing as compromised. 

UPDATE 1910: The restoration of the index files is progressing, but it's slow, as we are checking each index file to see if it has been compromised or simply replaced from a customer's own backup. If you have a backup / replacement index file and are having issues uploading it you may need to CHMOD 644 the current index.php
UPDATE 09:30 Friday Aug 20th

After 3 failed attempts at a full restore to a machine in our offices, we have successfully done a full restore to a machine in the data centre. This morning around 9am we restored any files which had a checksum that matched that of the defaced files that were placed there during the compromise on Saturday last.

Anyone who requires other files to be restored for any reason should contact us ASAP so we can restore them for you.

On the 11th of June we will be completing the upgrade of the core access switches in Interxion. As part of this, we need to physically move the current shared firewalls in Interxion to a different position within the rack. As they are a HA pair, it should be possible to do this without affecting connectivity.

The time line will be as follows:

02:00 Fail-over all the traffic to the second firewall in the pair. Once we're sure traffic is flowing through the second firewall, power down, move it to it's new position in the rack and recable. Power it back up and ensure that it's working as expected.

02:30 Repeat the procedure with the second firewall in the pair.

03:00 Install the new switches, and start swapping over from the current switches. There should be minimal downtime involved with this as each customer and rack switch has redundant connectivity back to the core.

The maintenance window will end at 06:00 once we're sure that every thing is back up and running as expected.

UPDATE Jun 12th 06:00 This maintenance window has now completed. Unfortunately not everything was completed, but the main work of moving the firewalls and getting the new switches in has been done. The few bit remaining will be completed during a future maintenance window.

We have been contacted by the IE Domain Registry in relation to DNS servers that are vulnerable to attack and exploit.

While the Blacknight DNS servers are NOT open to exploit using the "Kaminsky exploit" unfortunately a lot of other DNS servers are.

We will be attempting to contact as many of the affected registrants as possible to warn them of the issue.

If you want to test your DNS there is a free tool available here

VZWIN06 Hardware node needs to be rebooted at 8:45 tonight

 

This will effect the following VPSs:

78.153.208.255
78.153.208.163
78.153.210.20
78.153.210.160
78.153.210.172
78.153.209.204
78.153.210.184
78.153.209.166
78.153.210.203
78.153.210.81

We apologise for any inconvenience caused

Summary: Wordpress 2.8.4-3 now available to install and upgrade your Applications. This is a security release so please ensure you update your wordpress installs asap.

If you have any problems let us know.

We have noticed a very large number of user webspaces that have been attacked by the Gumblar virus (or a derivative of it)

If you are using a Windows based PC we urge you to:

• keep your antivirus software up to date
• scan your PC thoroughly

The virus acts by infecting a user's PC and then uses their FTP login details to install files on their website

NB: If your PC or that of one of your colleagues is infected you will need to clean it properly BEFORE you attempt to fix your website, as otherwise it will simply re-infect the site

UPDATE: Some people have reported success in using this web service to scan their sites for malicious code

UPDATE 2: Some more details on what Gumblar does on an infected PC and how it stops antivirus etc., from detecting it.

More information here