One of our shared windows servers - Palamedes - 81.17.248.55 has been rebooted now as some emergency Windows updates had to be applied to the server and a reboot for these to take effect.
The server should be back up fully in the next 20-30 minutes at the most, however the downtime may be less than this depending on how quickly the server can restart.
We apologise for any inconvenience this may cause you or your customers
The shared windows server - Palamedes 81.17.248.55 is having technical issues at the moment. The server has been rebooted fully and is in the process of restarting right now. Our engineers hope to have the machine fully back up in 20 - 30 minutes provided there are no hardware related problems.
This will affect all websites on this server, however any email traffic will remain completely un-effected.
We do apologise for any inconvenience this might be causing you or your customers.
Update 17:50: The server has been back up for some time now and our engineers can now confirm that everything is fine with the server and it is fully functioning again.
The maintenance work being conducted last night was completed last night and the servers moved to their new "home".
We were also able to use the downtime to install several patches and upgrade kernels etc. for both the shared hosting servers and the VPS nodes.
Three websites on three different shared servers were compromised by a hacker through weak FTP passwords. The hacker uploaded a trojan to these hosting packages and so these three servers were placed on anti-spam blacklists.
All three website owners have been contacted now and their FTP passwords reset. The offending files have been removed and the servers should be fully out of the blacklists soon. In the meantime for any users of the following servers they might be seeing some emails they send bouncing back to them undeliverable:
Galahad - 81.17.248.4
Gorlois - 81.17.252.85
Rivalin - 81.17.252.145
As a note to all users, please ensure all of your passwords are relatively secure. Some secure password tips would be:
# Don't use a dictionary word
# Don't use part of the username
# Keep the password at least 7 characters long
# Have a combination of at least three of:
- lowecase characters (a, b, c)
- uppercase characters (A, B, C)
- numbers (1, 2, 3)
- non-alphanumeric characters (!, %, *, {, £, )
Update (12.00pm): The three servers were removed from the blacklist about 90-120 minutes ago and most, if not all, mailservers around the world should have updated their blacklists to no longer include these three IP addresses. The IP addresses are fully removed from the blacklist itself.
Our engineers here have had to bring down the webservice (IIS) on the shared windows server Palamedes (81.17.248.55). This will cause your website to go down, however all email services will be unaffected.
The reason this service has been brought down is because it looks like many sites on the server were compromised last night and had their index pages defaced. In order to make the server secure and see how this hacker(s) compromised so many sites on the one server we have had to disable the webservice temporarily.
We hope to have this service enabled again as soon as possible and in the meantime we apologise for any inconveniences this may cause.
Update: 12:15
Currently we're working to delete all the files that contain the string of text that was put in place. This will cause many sites to show blank pages, but it'll also re-enable many sites on this server.
The restore is going to take some time to run as we can't filer out the index files. We don't have an estimated time to fix for everyone, but many of your .net apps will be back very shortly.
Update: 21:00 Sat August 9th
After a full days investigation we've found the hole that allowed this attack. A high profile site had an upload feature which allowed malicious attackers to upload arbitrary code. This code was an asp.net "shell" which was a basic web page which allowed the attackers access via to customers folders. We're unsure yet if there's any real protection in shared windows hosting regarding an attack vector like this, it's unlikely without restricting .net apps and causing functionality issues.
The site in question has been shut down and the owner contacted. We've also crawled through millions of files on the server to find any/all traces of the offensive index.html files placed on customers domains. We've also found some other copies of the ASP.net shell that the attackers left incase we found their primary entry point.
There will be one further update to this blog post in the coming week with further analysis of this attack vector and our solution to preventing it from happening again.
Final Update: Friday August 15 09:30
During our investigations of this intrusion we've noticed a few security implications. We've now taken measures to ensure that the default applcation pools for .net 1.1 and 2.0 do not run as the service Network Service. They now run as their own unprivileged users. We've also ensured that customers in their own application pools now run as their own web user also.
We've taken other measures to mitigate other attacks via other coding systems such as perl and php also to further strengthen the shared hosting platform.