Recently in hosted exchange Category

We are currently expierencing a Network issue which is affecting access to our Hosted Exchange platform.

Once we have an update we will post it here.

Update 16:45 Services are now restored.

Sorry for the interruption, we will investigate the issue to find the root cause

Summary: We're currently experiencing a network problem. We're working to diagnose it and get everything that is down backup.

Initial findings: We've found some MAC related log entries in some of our core network switches. These highlight a layer 2 loop within the network. These began @ 12:48. The knock on effect was layer 2 port flaps followed by BGP and OSPF flaps localised to that data centre. Any traffic passing through that data centre via Cogent, INEX LAN2 in or out would have been dead in the water.

At 12:56 the flaps stopped and the routers began to stabilise. It took a few minutes after this for everything to calm down. We're still investigating this right now and we'll post more information as it's available.

RFO: The reason for this outage was due to an ethernet loop within our network in the InterXion datacentre. A new piece of equipment was introduced to the network earlier this week fully configured and as such it caused no issues. While this equipment was being worked on it's configuration was wiped and upon reboot it appeared on the network and caused a network loop. This caused a spanning tree event in our core switching fabric in InterXion. The result was the network outage that was observed. Typically events like this can't occur and we put strict provisions in place to prevent it however in this instance a 3rd party piece of equipment caused the issue. It wasn't immediately evident that this would occur as the device in question had been tested in our lab for 4 weeks prior to it's deployment within the data centre. Obviously we take every precaution when working on our network and sometimes events like this can occur. However it should not happen in future and we're working on ways to prevent it happening again. At the very least we hope to contain issues to individual racks rather than the entire ethernet fabric in that data centre.

As part of our transition of our Hosted Exchange plans to Exchange 2010, we have had to whitelabel the autodiscovery service of existing Exchange 2007 customers.

Customers may receive a pop up warning them that https://exch2007.exchangehosting.eu/autodiscover/autodiscover.xml is trying to reconfigure their email settings in Outlook. This is normal behaviour and can be allowed.

Note: Hosted Exchange customers are reminded that if their DNS hosting is not using ns1.blacknight.com and ns2.blacknight.com then they must manually configure an autodiscover subdomain as follows: if your Hosted Exchange domain is domain.tld, then you must create a subdomain of domain.tld called autodiscover.domain.tld which must be created as a CNAME to autodiscover.blacknight.com

When: Monday August 16th from 02:00 until 8am

What: Control panel software, provisioning system, Agents on all hardware nodes, mysql nodes, web servers will all be upgraded from version 2.9.4 to version 5.0. During this window access to the control panel will be restricted. However e-mail, hosting etc services should not be affected by this upgrade.

Changes made to webspaces, new or existing e-mail accounts added or modified, new database creations etc during this window would be ill advised.

Services Affected:

cp.blacknight.com - Control Panel only.

Windows Shared hosting servers may be restarted. only if the upgrade of the management application proves to be problematic.

Linux Shared hosting servers may be restarted. only if the upgrade of the management application proves to be problematic.

Exchange Servers may be restarted, but due to the design of the Hosted Exchange no downtime should be noticed.

Qmail servers will have their imap/pop3/smtp services restarted as new versions of the software get put in place. Also changes may be made to LDAP so there could be intermittent authentication issues.

Sitebuilder servers should not be affected.

VPS nodes will have some software upgraded, but end users should experience no downtime.

Domain registrations and modifications will not be affected by this upgrade.

Update 07:40 Monday 16th:

The upgrade is still being performed at the moment, but it's in it's final stages. We'll post an "all clear" notice when it's complete.

Update 08:45 Monday 16th:

During the upgrade Parallels appear to have broken the provisioning system somehow. As this drives the Control Panel the CP is still down. They're working to resolve this issue but we've got no ETA as of yet.

Update 1045
The maintenance and upgrade has been completed. If anyone has any specific issues post-upgrade please contact our helpdesk

In order to add more redundancy into the network, we've moving certain segments of the network over to their own dedicated pairs of firewalls.

The following services will be affected:

• Miniumus, Medius and Maximus Windows and Linux shared hosted
  
• DirectAdmin shared hosting
• Windows Helm
• cp.blacknight.com
• Hosted Exchange
  

In each case, there should be minimal downtime as we're just shutting down the interface on the old firewalls and bringing it up on the new firewalls.

Update: 23:52

The above work is mostly completed. Due to time constraints we didn't complete it, however cp.blacknight.com and all our minimus/medius and maximus hosting packages have all been moved to the new firewall infrastructure.

During another maintenance window we'll complete this work, this current one is now closed.

As part of our ongoing commitment to bring you the latest in technology improvement, and as part of our overall transition to Exchange 2010, over the next 2 weeks we will be upgrading our Hosted Exchange platform to Exchange 2007 Service Pack 2.

As we designed the Hosted Exchange platform from the beginning to be completely resilient, there will be no downtime evident to Exchange users during this major upgrade.

At 23:00 (Irish time) tonight, Wednesday 12 August, we will be performing some essential maintenence on the Client Access servers in our Hosted Excange platform.

 

The only things effected will be client access to the platform - Outlook/Entourage/Outlook Anywhere/POP/IMAP. Mail will flow as normal during this window.

 

This maintenence window is for 10 minutes.

UPDATE: Apologies for not updating this sooner. The maintenance was conducted last night without issue

Summary: From 4am GMT tomorrow July 21st cp.blacknight.com will be down for short periods of time while we upgrade our cp software. This will continue until approx 09:00.

The following feature list will be available after this upgrade:

* PHP5 can now be run as an apache module or as php-cgi. This will allow better customisation of php variables via .htaccess. It also has some problems for customers running CMS applications however, so change this setting only if you know what you are doing.

* Perl on windows can now run as cgi or an isapi extension. Like php5 please only change if you know what you are doing.

* The ability to create temporary tables in MySQL should now be available to all users.

* php 4.4.9 and 5.2.9 will be supported on Linux shared hosting. php 5.2.9 will be supported on windows shared hosting.

* The linux file manager will be dropped in it's current form. A new integrated cp will be introduced which will reduce problems with ports, ssl mismatches etc. It will support unzipping and untaring .zip and .tar.gz files.

* DNS modification bugs have been fixed.

* several 100 bugs in various different sections of our CP have been fixed.

* php5 and 4 now have the mysql 5 client library compiled in so mysql access to mysql5 servers should work better than before. This will also elimiate the annoying notices that customers receive in phpmyadmin about library versions.

The above is only a summary of some of the major changes. There are going to be a lot of bug fixes as we've said along with other undocumented features.

Please let support know of any issues after 9am tomorrow.

update: 08:30 July 21st - important NOTEICE

The first phase of this upgrade has been completed. However seeing as it's a major version change there's already hotfix 1 available which fixes issues that occured post upgrade from 2.8 to 2.9 of our CP software.

I've authorised the install of hf01 to 2.9 this morning to ensure that the system is as stable as possible and that we don't encounter bugs today that are already fixed. The install of hf01 will take approx 2 hours so the cp will be up and down up until approx 11am.

UPDATE: 1214
The control panel is currently not accessible. We are awaiting an update from the software vendor

UPDATE 1250
The control panel should now be accessible and all other services should be working correctly. If there are any issues please let us know

UPDATE 1320
The control panel is currently experiencing some intermittent issues. The software vendors are dealing with this to get these issues resolved ASAP and we are awaiting an update.

UPDATE 1405
The control panel is available again, however there may be some lingering issues that the software vendors are still working on. Any further updates will be posted. If there are any issues you wish to make us aware of please let us know.

UPDATE 1433
If you are still having issues please email support@blacknight.com providing as much detail as possible so that we can reproduce the error

UPDATE 1501
There are intermittent issues with logging in to cp.blacknight.com - this matter is being looked into as a matter of urgency

UPDATE 2202
Cp.blacknight.com login issues appear to have been resolved for the last few hours. There have been several reports of other issues and our technical team is working through them. If you have any issues please let support@blacknight.com know and provide any details that you can

Apologies for any inconvenience that today's issues may have caused.

Summary: March 3rd @ 4am to 8am we'll be upgrading our new shared hosting platform and some back end services.

About 75 bugs are going to be fixed tomorrow morning starting at 4am. The result of this upgrade will mean that cp.blacknight.com and some other services will be down during the upgrade.

The maintenance window from parallels is approx 4 hours, so it should be fully completed by 8am.

A list of affected services and some well known bugs are below:

1) Hosted exchange bug fixes
2) linux shared hosting bug fixes
3) windows shared hosting bug fixes
4) shared e-mail, there will be some upgrades to our qmail servers which will cause pop3/imap issues during the maintenance window.
5) Bug#264931 upgradeSubscription for container if capacity VPS Number per HW node is over - currently if the limit we've set per hw node is reached and a customer gets an upgrade, their VPS will be shutdown and migrated to another node. This has been resolved.
6) Bug#271059 Webmail login does not work for 'webmail.yourdomain'


Update: March 3rd @ 09:45

This updated was delayed for a few hours but is currently being performed. It should be completed by 10:15 at the latest.

Update 2: 1253 - All services should have returned to  normal before 11am. However if you are experiencing issues that you think may be related please let us know.