When: Friday 12th and Saturday 13th of September Starting at 22:00 hours
What: All of the new shared hosting and VPS infrastructure, web servers, mail servers, database servers etc will be moved to a new cage in InterXion. This shouldn't result in extensive downtime, but it could be upto 2 hours.
During this time you won't be able to purchase domains, send e-mails, upload files or view/visit any sites on the platform. It'll take around 30 minutes to shut down all the machines and then take around 1 hour to move the cabs to a new cage.
If you have any queries about this please let us know ASAP.
Bors is currently experiencing some issues. We're working to resolve this as soon as possible.
Background:
Over the few days we've noticed the load on this server being above normal. We're investigating why and we think that the issue is being caused by comment spammers hammering some customers blogs. This is loading the mysql server on bors and causing the load to go up very quickly.
We hope to have a resolution to this issue today.
Our software vendors will be doing maintenance on the control panel tomorrow morning from around 7am to 8am.
During this period the control panel at cp.blacknight.com and the online shop will not be available.
This server went down at 13:09 and we're currently working to resolve the issue. Hopefully it'll be back promptly.
Update: 13:54
There appears to be a disk issue currently. This means that fsck is running on the machine at the moment which could take a long time as it's running on the /home partition which has circa 3million files or so on it. I will post another update when I thing I have an ETA for it to come back up.
Update: 19:30
We're still working on this server. We do apologise for this extended downtime and we assure you that once recovered from it, won't happen again as we're putting measures in place to mitigate such issues. To address concerns regarding e-mail, mostly sending mail servers will queue the e-mail for upto 5 days so you should all get your e-mail once it's back up.
Update: 21:45
All services should be restored. If anyone is still experiencing issues please contact our support desk
Tomorrow morning at 9am Parallels are upgrading our billing software to version 4.3.3 including Hotfix 01. This will result in our Store and CCP being down for a period of time. Services won't be affected during this time.
We'll be on hand before hand to make sure we're prepared and we'll test everything extensively to make sure there's no bugs after the install.
This will specifically affect only our new platform. i.e. https://store.blacknight.com and https://cp.blacknight.com
Update: 10:00 am August 14th
This hasn't happened yet as we had some complications on the test environment which we've been working on since 7am. I believe we've fixed this issue and we'll hopefully be upgrading the live environment shortly.
Expect the panel and the store to go offline for periods of time upto 11:30am
Update: 11:50 am August 14th
We've tested the upgrades and all appears well. This upgrade has fixed a bug with co.uk registrations and a few other bugs we'd been faced with.
Store and CP should be up and running completely now.
Our engineers here have had to bring down the webservice (IIS) on the shared windows server Palamedes (81.17.248.55). This will cause your website to go down, however all email services will be unaffected.
The reason this service has been brought down is because it looks like many sites on the server were compromised last night and had their index pages defaced. In order to make the server secure and see how this hacker(s) compromised so many sites on the one server we have had to disable the webservice temporarily.
We hope to have this service enabled again as soon as possible and in the meantime we apologise for any inconveniences this may cause.
Update: 12:15
Currently we're working to delete all the files that contain the string of text that was put in place. This will cause many sites to show blank pages, but it'll also re-enable many sites on this server.
The restore is going to take some time to run as we can't filer out the index files. We don't have an estimated time to fix for everyone, but many of your .net apps will be back very shortly.
Update: 21:00 Sat August 9th
After a full days investigation we've found the hole that allowed this attack. A high profile site had an upload feature which allowed malicious attackers to upload arbitrary code. This code was an asp.net "shell" which was a basic web page which allowed the attackers access via to customers folders. We're unsure yet if there's any real protection in shared windows hosting regarding an attack vector like this, it's unlikely without restricting .net apps and causing functionality issues.
The site in question has been shut down and the owner contacted. We've also crawled through millions of files on the server to find any/all traces of the offensive index.html files placed on customers domains. We've also found some other copies of the ASP.net shell that the attackers left incase we found their primary entry point.
There will be one further update to this blog post in the coming week with further analysis of this attack vector and our solution to preventing it from happening again.
Final Update: Friday August 15 09:30
During our investigations of this intrusion we've noticed a few security implications. We've now taken measures to ensure that the default applcation pools for .net 1.1 and 2.0 do not run as the service Network Service. They now run as their own unprivileged users. We've also ensured that customers in their own application pools now run as their own web user also.
We've taken other measures to mitigate other attacks via other coding systems such as perl and php also to further strengthen the shared hosting platform.
The VPS control panel will be unavailable for approximately 30 minutes from around 7am Irish time tomorrow morning (August 1st, 2008) while some minor patches are applied to the system.
This is to resolve a bug affecting functionality that was brought to light in the last couple of days
Tomorrow morning we are upgrading the billing part of our new platform. It should start around 6am Irish Time and last up until around 8am. During this time our CP may be down.
This update will fix several bugs we've found in the past couple of months and also bring some improvements that we've been waiting for.
The server Igraine - 81.17.252.25 is currently experiencing a delay in emails going through is due to high activity on the server, and a large baklog of email.
Our engineers are looking into resolving the situation as quickly as possible now and you should notice the delay get smaller as the server works through the backlog.
UPDATE: This issue has now been fully resolved. It looks like there was an out of date install of PHPBB on a customer's site that was being abused by spammers and this was causing the high load and mail backlog.