Summary: Tonight Monday March 1st we're moving the Provisioning node known as OSSCORE to new hardware. This is due to the sheer volume of customers going into the system on a daily basis. This has caused our provisioning back end to get slower and slower which is causing cp.blacknight.com (web hosting) to slow down.
When: March 1st 2010 starting at 22:00 until 01:00 on March 2nd.
What: From 22:00 hours we'll be working on the migration, the cp on cp.blacknight.com will be turned off for the duration of this migration in order to prevent inconsistencies
Services affected: Management of your web hosting plans, e-mail, databases etc will not be available during this window but your hosting will be unaffected.
Update 23:00: this window has been completed successfully a full 2 hours early.
Ragnell is currently having issues which is causing it to become unresponsive. There is an engineer on site getting it backup, and it should reappear within the next 10mins.
At around 15:30 balin.blacknight.ie stopped responding to requests. As we had an engineer on site who also couldn't log in locally, we rebooted it straight away and it was back up by 15:40.
While we don't know exactly what caused the issue yet, it looks like server was run out of memory, possibly due to a massive surge of queries to MySQL.
Summary: Tomorrow morning starting at 04:00 our billing system bound within the "billing" tab of cp.blacknight.com is going to be upgraded by 2 minor revisions.
Details: During the upgrade our billing system will be unavailable. This means that domain and hosting purchases, upgrades, licence upgrades and domain contact, name server management will all be unavailable. The upgrade shouldn't take longer than about 2 hours so it will be finished by 6am.
During this window we'll be installing a new build of our own registrar and payment plugins. The payment plugin has no updates but the registrar plugin has some fixes that we've been waiting to deploy.
Summary: On Thursday November 26th we will be having POA 2.9 HF03 installed on our control panel. This has some feature improvements and also some bug fixes in relation to DNS propagation delays. There are also some other minor bugs that will be fixed.
When: 4am Thursday November 26th
What: Major software update of our control panel and provisioning software. This long awaited upgrade fixes a number of bugs we've encountered since the last upgrade on July 21st this year.
This means that from around 4am Irish time until Circa 8am Irish time our control panel cp.blacknight.com will be offline. Your hosting, e-mail and other ancillary services shouldn't be affected bar a restart of iis and apache post upgrade. These should all be staged and shouldn't cause more than a minute or twos down time for your websites.
If after 9am on Thursday you're experiencing any difficult please let us knows.
Summary: Wordpress 2.8.4-3 now available to install and upgrade your Applications. This is a security release so please ensure you update your wordpress installs asap.
If you have any problems let us know.
When: Wednesday September 16th at 4am
What: Control panel and backend service upgrade to POA 2.9 HF02 which will prevent access to cp.blacknight.com for the duration of the upgrade.
Overview: We're currently running quite a new version of POA 2.9 and there are a number of bugs in this version. We'll be upgrading to POA HF02 which will fix a number of UI, provisioning and general system bugs that crept into our system since the last upgrade around 6 weeks ago. There's also a couple of new features introduced however these won't affect our existing customers for the time being.
Update: This update is a reminder that this maintenance window is going ahead tomorrow morning.
Update: 09:45
There are two issues since the upgrade this morning that are causing people issues.
Firstly webmail.$domain.tld isn't currently working. As a work around, you can login to https://webmail.blacknight.com and use the mail server "mail.blacknight.com" with your normal e-mail address and password.
Secondly all webspaces with php5 as apache mode seem to have somehow reverted to php4. We understand that this is a very serious issue and we've raised it with the highest possible priority with Parallels to get it fixed. The only short term work around is to change your webspace to use php5 cgi mode, but this won't suit everyone so it's not recomended unless your site is badly effected.
Update: 16:25 - all issues resolved and the upgrade ticket is now closed.
As of around midday the above issues were resolved. We were just waiting to see if any other issues cropped up before we gave the all clear on this upgrade.
Summary: Today at 16:39 customers on several firewall segments were impacted by high packetloss. This was the result of a yet unknown event, however we were able to restore service within a timely manner by forcing the "active" firewall into standby thus forcing the standby firewall to take over.
We're currently investigating the cause further and examining syslogs, port graphs, cpu graphs etc. We will post a conclusion to the issue once our engineering team have completed their investigations.
Current situation as of 17:10:
Network connectivity was restored within 10 minutes of the actual event occuring however some services like e-mail on our shared hosting cluster will take some time to recover as there are an extraordinary number of inbound connections into the cluster and the load balancer is doing it's best to diviy out connections to the mail servers.
Update 1810
Mail services should be back to normal, but if there are any issues let us know.
