Notification Type
Security Notification
Service Affecting
No
Message
Sometime before midnight last night, morgana was compromised with a root level exploit. It looks like any index file has been overwritten on all sites. Mail seems to be unaffected.The most recent uncompromised backup seems to be 16:00 yesterday, so we're working to restore the overwritten html files from that.
When we've more data available, it will be put up here.
Update 15:30: Mail is back up and running, however we're running into delays restoring compromised index pages from the backups. The backups are fine, however the program for restoring them seems to be sulking due the presence of dead symlinks in the backup.
We are working with the vendors to resolve this and get a full restore. In the meantime, we're going to use a previously restored backup from Oct18th to get sites up and running again. Once the bug with more current restore is resolved, we will bring back up yesterday's versions of the files.
Update 01:30: The mass restore of files from Oct 18th is going at full swing and should be completed within an hour or two.
Update: 08:30 Monday Dec 6th.
The mass restore is taking a little longer than anticipated. it's working it's way through alphabetically currently. This will take probably another 16-24 hours but we might stop it and run it in a threaded fashion instead of a single thread which will be much faster.
There have been a large number of sites restored to their state from the 4th of December. However as index* in terms of filenames were overwritten there's many sub directories of websites not working.
Please open a support ticket and our support team can restore individual files for you. However we will ask for some patience as we want the mass restore to finish before we start doing further restores.
We are still working with the vendor in order to get the restore of yesterday's backups running properly. We do have a full backup, and we can easily restore any individual files or directories if there's something badly needed, however a mass restore like we can do on the backup from Oct 18th is failing.
December 6th @ 12:00
The restore of the full backup from October 18th 2010 has completed. The full CDP snapshot from December 4th @ 15:21 is at 65.7 GB of 111.1 GB. This will take another 2hours or so to restore at which point we're going to completely restore all public_html and private_html directories completely to this date/time.
If you have critical sites which are still down please contact us immediately via our helpdesk, however until this full restore is complete we can't access this snapshot so we can't do individual file restores.
December 6th @ 14:30
The restore from the back of Dec 4th @ 15:21 has now completed. This has been restored to another server. We are now proceeding with the restoration of changed files using rsync. This will take several hours to complete and we will update this post again once we're done.
We are now in a position to restore individual files on request from people so please let us know if there is something urgent that you need done.
December 6th @ 16:45
The restore of files from December 4th should now be complete. This issue is considered closed. If you're having any problems please let us know and we can restore files manually for you very quickly.
I know one or more of my websites uses morgana, but I have no direct way of knowing which, it's just that I remember the name.
Can you notify those customers whose index files were changed, and what they were changed to overnight?
Thanks
Dermod
PS can't login using OpenId, google or blogspot. pain in the *)*)
Is it possible that we can FTP our own back-ups, or at least let us know when we can access to servers again ?
When are you guys going to secure your servers properly? This same problem has happened on more than one occasion, and it takes you days to resolve. Unacceptable.