This server is currently experiencing difficulties serving HTTP traffic.
We are aware of the issue, and hope to have a resolution as soon as possible.
Update 10:25 - All services have returned to normal.
Summary: We're rebooting pemvzmps16 NOW (11:30am, Friday 27th) to bring it back up on the correct kernel after a memory upgrade last night.
Details: The hw node that these two VMs reside on had a failed disk replaced last night. As a result a new kernel was loaded. We've identified that the kernel that is currently loaded is the incorrect kernel and is causing some load issues on these two nodes. A reboot onto the correct kernel will resume normal operations.
Update 11:37: this is now complete and normal service has been restored.
Details: The hw node that these two VMs reside on had a failed disk replaced last night. As a result a new kernel was loaded. We've identified that the kernel that is currently loaded is the incorrect kernel and is causing some load issues on these two nodes. A reboot onto the correct kernel will resume normal operations.
Update 11:37: this is now complete and normal service has been restored.
We have been alerted to a failed hard disk on one of our linux shared hosting nodes PEMVZMPS16. To ensure the RAID becomes full optimal asap we are going to replace this disk tonight at 9PM.
What's affected?
Two linux shared hosting web servers are located on this node. They are:
The plan is to bring the server offline, replace the disk, boot it back up and then let the RAID resync itself.
The downtime will be in the timeframe of 15 minutes, but we are going to schedule a window of one hour.
When is this happening ?
The downtime will occur tonight 26th of August 2010 at 21:00 hours.
Once the maintenance is complete we'll update this blog post.
UPDATE 21:08 - The disk change is complete and the server is now back online.
What's affected?
Two linux shared hosting web servers are located on this node. They are:
- 81.17.254.74 pemlinweb15.blacknight.com
- 81.17.254.75 pemlinweb16.blacknight.com
The plan is to bring the server offline, replace the disk, boot it back up and then let the RAID resync itself.
The downtime will be in the timeframe of 15 minutes, but we are going to schedule a window of one hour.
When is this happening ?
The downtime will occur tonight 26th of August 2010 at 21:00 hours.
Once the maintenance is complete we'll update this blog post.
UPDATE 21:08 - The disk change is complete and the server is now back online.
We are currently experiencing issues with two linux shared hosting nodes:
pemlinweb32.blacknight.com - 81.17.254.44
pemlinweb33.blacknight.com - 81.17.254.48
Our engineerings are working to resolve this issue asap and will keep this blog post updated.
UPDATE 12:20AM - This issue has been resolved.
There has been a slight change in how the email accounts work since our major cp.blacknight.com upgrade on Monday the 16th of August:
http://www.blacknightstatus.com/2010/08/cpblacknightcom-major-upgrade.html
This will not affect the vast majority of customers but if you are having problems emailling an address on your domain that worked fine before Monday, but now no longer works, this might be the cause.
How the email accounts work is they have two parts: a service user and an email address connected to that service user. The service user is your username and password that you use to access the email address itself. While we would always recommend the service user have the same username as your email address, you might not have set up your email account this way.
Prior to Monday your email account would accept email sent to either the service user username or the email address. So for example if your username was info@test.com but the email account associated with this was actually john@test.com then emails to info@test.com would still get delivered to the john@test.com address without error.
This was not actually how the system should work however and the upgrade on Monday fixed this "bug". Now email will only go the email account itself and email to, for example, info@test.com would no longer get through. You can resolve this easily for yourself now as follows:
# http://cp.blacknight.com > Email > Email Addresses
# Click on the affected service user name in the list
# Go to Email Addresses > Add > Add the email address that no longer works > Submit
That should start to work again straight away. If you continue to have problems, or have any questions, please let us know as soon as possible.
http://www.blacknightstatus.com/2010/08/cpblacknightcom-major-upgrade.html
This will not affect the vast majority of customers but if you are having problems emailling an address on your domain that worked fine before Monday, but now no longer works, this might be the cause.
How the email accounts work is they have two parts: a service user and an email address connected to that service user. The service user is your username and password that you use to access the email address itself. While we would always recommend the service user have the same username as your email address, you might not have set up your email account this way.
Prior to Monday your email account would accept email sent to either the service user username or the email address. So for example if your username was info@test.com but the email account associated with this was actually john@test.com then emails to info@test.com would still get delivered to the john@test.com address without error.
This was not actually how the system should work however and the upgrade on Monday fixed this "bug". Now email will only go the email account itself and email to, for example, info@test.com would no longer get through. You can resolve this easily for yourself now as follows:
# http://cp.blacknight.com > Email > Email Addresses
# Click on the affected service user name in the list
# Go to Email Addresses > Add > Add the email address that no longer works > Submit
That should start to work again straight away. If you continue to have problems, or have any questions, please let us know as soon as possible.
Eurid, the registry operator for .eu, will be conducting maintenance on August 25th 2010 from 0600 to 0700 CEST.
During this period we will not be able to process any new registrations or updates.
Existing domain names will not be affected.
The control panel is currently unavailable.
More information as soon as we have it
1230 Update: It looks like there is some configuration issue presumably related to yesterday's upgrade. We have contacted the software vendors and hope to have an update and resolution quickly
Update 1243 - We have restarted parts of the system, so the control panel login is currently available. We will update when we have more information
Our shared hosting server Ragnell has been compromised, and the majority of the index.php's have been replaced with a hacked version. We have disabled all copies of the compromised index files already.
We are at the moment making sure the hole used is fixed before re-enabling Apache. As part of this, PHP is being upgraded to PHP5.
We are also going to see about restoring the disabled index files, however this is going to take longer. The backup system we use is geared towards full system backups, so restoring individual files is likely to take a while. If you have an uptodate copy of your index file, it will probably be faster if you get it uploaded yourself. This can be done even while Apache is down.
Update 1430: The upgrade of php / Apache is almost complete. Once it's finished we will be able start restoring index files from backups.
Update 1515: Apache is back up and running. We are currently restoring the index files from backup. This is going to take a long time.
UPDATE 1615: If your site's index file has been restored or if you've restored it yourself let us know if there are any issues.
UPDATE 16:52: As restoring individual index files is proving to be far too unwieldy, we are currently restoring the whole partition to another box. This will allow us to script the restore of any index files which are still showing as compromised.
UPDATE 1910: The restoration of the index files is progressing, but it's slow, as we are checking each index file to see if it has been compromised or simply replaced from a customer's own backup. If you have a backup / replacement index file and are having issues uploading it you may need to CHMOD 644 the current index.php
UPDATE 09:30 Friday Aug 20th
After 3 failed attempts at a full restore to a machine in our offices, we have successfully done a full restore to a machine in the data centre. This morning around 9am we restored any files which had a checksum that matched that of the defaced files that were placed there during the compromise on Saturday last.
Anyone who requires other files to be restored for any reason should contact us ASAP so we can restore them for you.
We are at the moment making sure the hole used is fixed before re-enabling Apache. As part of this, PHP is being upgraded to PHP5.
We are also going to see about restoring the disabled index files, however this is going to take longer. The backup system we use is geared towards full system backups, so restoring individual files is likely to take a while. If you have an uptodate copy of your index file, it will probably be faster if you get it uploaded yourself. This can be done even while Apache is down.
Update 1430: The upgrade of php / Apache is almost complete. Once it's finished we will be able start restoring index files from backups.
Update 1515: Apache is back up and running. We are currently restoring the index files from backup. This is going to take a long time.
UPDATE 1615: If your site's index file has been restored or if you've restored it yourself let us know if there are any issues.
UPDATE 16:52: As restoring individual index files is proving to be far too unwieldy, we are currently restoring the whole partition to another box. This will allow us to script the restore of any index files which are still showing as compromised.
UPDATE 1910: The restoration of the index files is progressing, but it's slow, as we are checking each index file to see if it has been compromised or simply replaced from a customer's own backup. If you have a backup / replacement index file and are having issues uploading it you may need to CHMOD 644 the current index.php
After 3 failed attempts at a full restore to a machine in our offices, we have successfully done a full restore to a machine in the data centre. This morning around 9am we restored any files which had a checksum that matched that of the defaced files that were placed there during the compromise on Saturday last.
Anyone who requires other files to be restored for any reason should contact us ASAP so we can restore them for you.
The COCCA registry platform, which is used by .fm, .gs, .cx, .ht and several other country code top level domains, has scheduled maintenance on August 18th
When?
August 18th 2300 - 2330
What is impacted?
New registrations, updates and lookups for .fm, .gs and other country codes.
Existing .fm domain names will not be impacted
When: Monday August 16th from 02:00 until 8am
What: Control panel software, provisioning system, Agents on all hardware nodes, mysql nodes, web servers will all be upgraded from version 2.9.4 to version 5.0. During this window access to the control panel will be restricted. However e-mail, hosting etc services should not be affected by this upgrade.
Changes made to webspaces, new or existing e-mail accounts added or modified, new database creations etc during this window would be ill advised.
Services Affected:
cp.blacknight.com - Control Panel only.
Windows Shared hosting servers may be restarted. only if the upgrade of the management application proves to be problematic.
Linux Shared hosting servers may be restarted. only if the upgrade of the management application proves to be problematic.
Exchange Servers may be restarted, but due to the design of the Hosted Exchange no downtime should be noticed.
Qmail servers will have their imap/pop3/smtp services restarted as new versions of the software get put in place. Also changes may be made to LDAP so there could be intermittent authentication issues.
Sitebuilder servers should not be affected.
VPS nodes will have some software upgraded, but end users should experience no downtime.
Domain registrations and modifications will not be affected by this upgrade.
Update 07:40 Monday 16th:
The upgrade is still being performed at the moment, but it's in it's final stages. We'll post an "all clear" notice when it's complete.
Update 08:45 Monday 16th:
During the upgrade Parallels appear to have broken the provisioning system somehow. As this drives the Control Panel the CP is still down. They're working to resolve this issue but we've got no ETA as of yet.
Update 1045
The maintenance and upgrade has been completed. If anyone has any specific issues post-upgrade please contact our helpdesk
