Our Ahsay backup install will be upgraded to the latest version on Monday. Access to the web interface will be blocked while the upgrade is in progress. Downtime should be under 30 minutes.
September 2009 Archives
Summary: At 09:04 we had a 30 second blip in InterXion. This happened during the process of bringing a series of new switches online. Ordinarily this operation is non service affecting however for some reason it caused a problem on a number of vlans.
All service was restored on or before 09:05 and the case is closed.
All service was restored on or before 09:05 and the case is closed.
We are currently experiencing a network DDoS attack. We are working to block the attack asap and will update this post with regular status updates.
Update 11.05: The situation is currently back to normal operations, the issue has been blackholed and the routers are showing normal loads again. The engineering team is monitoring the network closely.
Update 11.41 The issue has been tracked back to one of our peers causing floods of traffic on our transit routers. Engineers are currently blacklisting the errorous traffic.
Update 11.49 The network has stabilized again. We are monitoring it closely.
Update 11.05: The situation is currently back to normal operations, the issue has been blackholed and the routers are showing normal loads again. The engineering team is monitoring the network closely.
Update 11.41 The issue has been tracked back to one of our peers causing floods of traffic on our transit routers. Engineers are currently blacklisting the errorous traffic.
Update 11.49 The network has stabilized again. We are monitoring it closely.
From approximately 2115 last night until appoximately 1055 this morning there were issues affecting the IEDR's backend systems
These issues resulted in updating DNS servers for IE domains. It may also have led to delays in conducting other maintenance activities on IE domains
The issue has now been resolved
![Reblog this post [with Zemanta]](http://img.zemanta.com/reblog_e.png?x-id=dd109cbf-d3b1-4741-91c7-236b9db38026)
These issues resulted in updating DNS servers for IE domains. It may also have led to delays in conducting other maintenance activities on IE domains
The issue has now been resolved
We will be doing some maintenance on our core network in Interxion. We have to move some connections in order to facilitate the introduction of extra firewalls at a future date.
As all the connections are redundant, there should no interruption to the network, however there could be two slight blips are the network figures out the new paths.
Update 26/09/09 07:05: This work has been completed with no interruption to the network.
As all the connections are redundant, there should no interruption to the network, however there could be two slight blips are the network figures out the new paths.
Update 26/09/09 07:05: This work has been completed with no interruption to the network.
Summary: Wordpress 2.8.4-3 now available to install and upgrade your Applications. This is a security release so please ensure you update your wordpress installs asap.
If you have any problems let us know.
If you have any problems let us know.
When: Wednesday September 16th at 4am
What: Control panel and backend service upgrade to POA 2.9 HF02 which will prevent access to cp.blacknight.com for the duration of the upgrade.
Overview: We're currently running quite a new version of POA 2.9 and there are a number of bugs in this version. We'll be upgrading to POA HF02 which will fix a number of UI, provisioning and general system bugs that crept into our system since the last upgrade around 6 weeks ago. There's also a couple of new features introduced however these won't affect our existing customers for the time being.
Update: This update is a reminder that this maintenance window is going ahead tomorrow morning.
Update: 09:45
There are two issues since the upgrade this morning that are causing people issues.
Firstly webmail.$domain.tld isn't currently working. As a work around, you can login to https://webmail.blacknight.com and use the mail server "mail.blacknight.com" with your normal e-mail address and password.
Secondly all webspaces with php5 as apache mode seem to have somehow reverted to php4. We understand that this is a very serious issue and we've raised it with the highest possible priority with Parallels to get it fixed. The only short term work around is to change your webspace to use php5 cgi mode, but this won't suit everyone so it's not recomended unless your site is badly effected.
Update: 16:25 - all issues resolved and the upgrade ticket is now closed.
As of around midday the above issues were resolved. We were just waiting to see if any other issues cropped up before we gave the all clear on this upgrade.
What: Control panel and backend service upgrade to POA 2.9 HF02 which will prevent access to cp.blacknight.com for the duration of the upgrade.
Overview: We're currently running quite a new version of POA 2.9 and there are a number of bugs in this version. We'll be upgrading to POA HF02 which will fix a number of UI, provisioning and general system bugs that crept into our system since the last upgrade around 6 weeks ago. There's also a couple of new features introduced however these won't affect our existing customers for the time being.
Update: This update is a reminder that this maintenance window is going ahead tomorrow morning.
Update: 09:45
There are two issues since the upgrade this morning that are causing people issues.
Firstly webmail.$domain.tld isn't currently working. As a work around, you can login to https://webmail.blacknight.com and use the mail server "mail.blacknight.com" with your normal e-mail address and password.
Secondly all webspaces with php5 as apache mode seem to have somehow reverted to php4. We understand that this is a very serious issue and we've raised it with the highest possible priority with Parallels to get it fixed. The only short term work around is to change your webspace to use php5 cgi mode, but this won't suit everyone so it's not recomended unless your site is badly effected.
Update: 16:25 - all issues resolved and the upgrade ticket is now closed.
As of around midday the above issues were resolved. We were just waiting to see if any other issues cropped up before we gave the all clear on this upgrade.
We are currently experiencing some issues with one of our shared hosting servers, Gorlois.
We are troubleshooting the issue currently and will update the status blog as soon as we have more information.
Update 11:19: The server is back online after a reboot. We are now looking into the cause of the issue.
We are troubleshooting the issue currently and will update the status blog as soon as we have more information.
Update 11:19: The server is back online after a reboot. We are now looking into the cause of the issue.
Due to some disk inconsistencies we are currently seeing across two hardware nodes we are migrating 4 mysql servers to new hardware nodes this evening.
The maintenance will be taking place at 23:00. MySQL will be inaccessible during this time to ensure that data is consistent when migrating the MySQL nodes.
The estimated downtime for this window is 15 minutes per MySQL node.
The MySQL nodes affected are:
mysql357.cp.blacknight.com
mysql358.cp.blacknight.com
mysql359.cp.blacknight.com
mysql360.cp.blacknight.com
Once all work is complete an update will be posted.
Thank you for your understanding.
UPDATE 23:45: All maintenance was completed as expected and without any issues.
The maintenance will be taking place at 23:00. MySQL will be inaccessible during this time to ensure that data is consistent when migrating the MySQL nodes.
The estimated downtime for this window is 15 minutes per MySQL node.
The MySQL nodes affected are:
mysql357.cp.blacknight.com
mysql358.cp.blacknight.com
mysql359.cp.blacknight.com
mysql360.cp.blacknight.com
Once all work is complete an update will be posted.
Thank you for your understanding.
UPDATE 23:45: All maintenance was completed as expected and without any issues.