Recently in email Category

Summary: mail.blacknight.com and it's associated services are running slower than normal today due to increased activity from end users. This increase in activity is not part of the normal pattern of use so we're trying to pinpoint the cause.

Services affected: smtp, imap and pop3.

ETA For a fix: Approx 1 hour

Update: Jan 18th @ 12:41 pm

We've switched storage appliances on the back end of the mail cluster. E-mail is out of sync by 1 day but it's currently synchronising. As a result of this mail auth, and general mail delivery, connection times have decreased dramatically.

You may get some duplicated e-mail, but this was preferable over e-mail being down for several hours while we fight with the current storage platform.

Our technical team is currently working on an issue impacting our mail cluster.

The directory services are having some issues, so authentication is currently not working

In simple terms - people cannot login to collect or send email

If you need to send email urgently please change your email SMTP to that of your ISP (broadband provider)

We will update you as soon as we know more.

Services affected: webmail, pop, smtp, IMAP

UPDATE 2137
We have been running the cluster in a degraded state for the last 45 minutes approximately and are monitoring the situation.

UPDATE December 2nd 1426
Apologies for not updating earlier
We have been monitoring this situation closely and the cluster appears to be handling authentication without errors. If you do, however, have any issues please let support know.

We are currently experiencing some intermittent issues with our mail cluster located at mail.blacknight.com

Our engineers are currently working to resolve this issue asap.

We will update this blog post once operations are back to normal.

UPDATE 16:16: The erroneous server has been removed from the mail cluster completely. This means all services are back to normal but missing some power. We are working to get the server back online.

UPDATE 16:44: The server has no been brought online and introduced back into the cluster.

UPDATE 17:06: Due to inconsistencies with ldap, we are trying to rebuild the index. This is causing some issues currently which we are working on now. Please bare with us.

UPDATE 17:10: Mail is being routed correctly now. We are monitoring it closely.

UPDATE Nov 25th 11:30: There's still some lingering issues with the mail cluster which are causing POP timeouts. We're currently investigating and working with our vendors in order to try and narrow down exactly where the problem is occuring.

UPDATE Nov 25th 12:40: This issue is on going. There appears to be a locking issue on the nfs server which is being caused by a huge volume of pop3 connections coming into the servers.

UPDATE Nov 25th 15:00: This issue is still on going. We're aware 1000s of pop3 and imap users can not currently access e-mail. The problem is still be worked on and we hope to have services restored today. We may need to put an emergency maintenance window in place in order to fully diagnose the issue. Right now the engineering team is tracing the high load on the mail servers back to the NFS file server and the storage it uses.

The storage is provided by an EMC Clariion CX300 Fibre Channel SAN. This should facilitate 100s of thousands of concurrent users however it seems that at extremely busy times it can't cope. The main reason for this appears to be the fact that the pop3 process immediately does a read on Maildir folder for the mailbox that is logging in without the list command being sent to by the client application. That means that every single auth command creates an IO intensive job that the mail servers have to deal with. We're doing everything within our power to resolve this and we've put a new storage array into the data centre. We hope to bring this online tonight. During which time we'll have to take mail completely offline.

Points of Interest:

1) Both pop3 and imap are affected by this issue.
2) Delivery of inbound e-mail to your inbox is unaffected.
3) if we do decide on a maintenance window, we'll store all e-mail off the cluster and forward it on once the qmail cluster has returned to good health.
4) During the move to the new system, your old e-mail may dissappear, however it'll arrive back in your imap folder within 24-48 hours as we sync data back from the old storage server.
5) we're working with Parallels to devise a far better storage arrangement and configuration of the qmail cluster. This has been on going for a couple of months but we believe there is an end in sight.
6) Our support team are trying to deal with all your calls, e-mails and live chats as quickly as possible. Please bear in mind that the problem is out of their control and that the most up to date information on the issue will be posted here on the status blog.

Services affected currently: Webmail, imap and pop mail - smtp should continue as normal.

UPDATE Nov 26th 09:04: Services are resumed but at a degraded rate currently. Our engineers are currently executing a plan of action to get the system back up and running at full speed again. Please bare in mind the sheer volume of data we are working with, this is the primary factor on why this issue is taking some time to resolve.

Our Shared Hosting Qmail cluster is currently experiencing some problems. This has resulted in some people experiencing timeouts collecting their emails.

 

More on this as we find out more.

12:35 Tuesday Oct 27th:

Currently some customers are experiencing issues collecting e-mail from our pop3/imap cluster. We seem to be having some memory issues which are causing connections drops by Courier-IMAP which handles all inbound imap/pop3 connections.

12:53 Tuesday Oct 27th:

This issue is resolved. It's again down to a process that we've no control over causing problems on our mail cluster. We'll investigate this further with Parallels and see can we get a permanent resolution to our problem.

18:40 Tuesday Oct 27th:

We've disabled the whosond service on our qmail cluster. This is causing some e-mail issues which are not really issues per se but they could be perceived as such. In order to configure your e-mail client to use SMTP Authentication please look at the following KB article:

https://support.blacknight.ie/472/smtp-authentication.html

Summary: Most heavy users on our system have been experiencing some slowdown of e-mail for the last couple of weeks. This was up until recently intermittent however it is now a problem that occurs during peak usage times. i.e. between 11am and 5pm or so.

The cause for this issue appears to be because of a number of factors, some of which are out of our control.

Within our control our storage array that is well speced doesn't appear to be doing the business. i.e. under heavy slow it's quite slow to respond. We've taken this up with the vendor in question, however we're building a second SAN to replace it. We hope to put the new storage in place during the weekend of October 17 and 18th. This should give much better performance over all and will also stop the mail delays.


Outside of our control we have ldap and whosond which are services parallels use to maintain certain databases of user information that qmail and couier-imap use to get information about users. Each time these have issues it causes problems with mail delivery, this compounded by the storage array being slow is the cause of the problems.

We've also got ram upgrades which we're going to put in place tonight on all the mail nodes,
because we're clustering them there'll be minimal impact during the ram upgrades.

Please bare with us while we get this issue resolved.

We have been experiencing issues with our mail cluster for the last few minutes.

At 12:52 SMTP services started failing on the cluster. Once our engineers located the issue within the cluster, normal services were resumed at 13:01PM

If you are still experiencing any issues with mail, please contact support asap.

At 23:00 (Irish time) tonight, Wednesday 12 August, we will be performing some essential maintenence on the Client Access servers in our Hosted Excange platform.

 

The only things effected will be client access to the platform - Outlook/Entourage/Outlook Anywhere/POP/IMAP. Mail will flow as normal during this window.

 

This maintenence window is for 10 minutes.

UPDATE: Apologies for not updating this sooner. The maintenance was conducted last night without issue

Summary: From 4am GMT tomorrow July 21st cp.blacknight.com will be down for short periods of time while we upgrade our cp software. This will continue until approx 09:00.

The following feature list will be available after this upgrade:

* PHP5 can now be run as an apache module or as php-cgi. This will allow better customisation of php variables via .htaccess. It also has some problems for customers running CMS applications however, so change this setting only if you know what you are doing.

* Perl on windows can now run as cgi or an isapi extension. Like php5 please only change if you know what you are doing.

* The ability to create temporary tables in MySQL should now be available to all users.

* php 4.4.9 and 5.2.9 will be supported on Linux shared hosting. php 5.2.9 will be supported on windows shared hosting.

* The linux file manager will be dropped in it's current form. A new integrated cp will be introduced which will reduce problems with ports, ssl mismatches etc. It will support unzipping and untaring .zip and .tar.gz files.

* DNS modification bugs have been fixed.

* several 100 bugs in various different sections of our CP have been fixed.

* php5 and 4 now have the mysql 5 client library compiled in so mysql access to mysql5 servers should work better than before. This will also elimiate the annoying notices that customers receive in phpmyadmin about library versions.

The above is only a summary of some of the major changes. There are going to be a lot of bug fixes as we've said along with other undocumented features.

Please let support know of any issues after 9am tomorrow.

update: 08:30 July 21st - important NOTEICE

The first phase of this upgrade has been completed. However seeing as it's a major version change there's already hotfix 1 available which fixes issues that occured post upgrade from 2.8 to 2.9 of our CP software.

I've authorised the install of hf01 to 2.9 this morning to ensure that the system is as stable as possible and that we don't encounter bugs today that are already fixed. The install of hf01 will take approx 2 hours so the cp will be up and down up until approx 11am.

UPDATE: 1214
The control panel is currently not accessible. We are awaiting an update from the software vendor

UPDATE 1250
The control panel should now be accessible and all other services should be working correctly. If there are any issues please let us know

UPDATE 1320
The control panel is currently experiencing some intermittent issues. The software vendors are dealing with this to get these issues resolved ASAP and we are awaiting an update.

UPDATE 1405
The control panel is available again, however there may be some lingering issues that the software vendors are still working on. Any further updates will be posted. If there are any issues you wish to make us aware of please let us know.

UPDATE 1433
If you are still having issues please email support@blacknight.com providing as much detail as possible so that we can reproduce the error

UPDATE 1501
There are intermittent issues with logging in to cp.blacknight.com - this matter is being looked into as a matter of urgency

UPDATE 2202
Cp.blacknight.com login issues appear to have been resolved for the last few hours. There have been several reports of other issues and our technical team is working through them. If you have any issues please let support@blacknight.com know and provide any details that you can

Apologies for any inconvenience that today's issues may have caused.

At around 14:40 today we began to notice a slow down in connections to parts of our firewalled network in the InterXion data centre.  This can affect any of our new shared hosting plans, VPS plans, and some dedicated and colocated equipment.

Our engineers are working on what may be causing the network congestion and the symptoms range from either a slow connection to your website, VPS, or server, or a temporary loss in connection entirely. 

This is not affecting all services in this data centre and it is not a total loss of connectivity, there is just congestion that is slowing down the traffic into some parts of the firewalled network that protect the above systems.