Recently in email Category

There has been a slight change in how the email accounts work since our major cp.blacknight.com upgrade on Monday the 16th of August:

http://www.blacknightstatus.com/2010/08/cpblacknightcom-major-upgrade.html

This will not affect the vast majority of customers but if you are having problems emailling an address on your domain that worked fine before Monday, but now no longer works, this might be the cause.

How the email accounts work is they have two parts: a service user and an email address connected to that service user.  The service user is your username and password that you use to access the email address itself.  While we would always recommend the service user have the same username as your email address, you might not have set up your email account this way.

Prior to Monday your email account would accept email sent to either the service user username or the email address.  So for example if your username was info@test.com but the email account associated with this was actually john@test.com then emails to info@test.com would still get delivered to the john@test.com address without error.

This was not actually how the system should work however and the upgrade on Monday fixed this "bug".  Now email will only go the email account itself and email to, for example, info@test.com would no longer get through.  You can resolve this easily for yourself now as follows:

# http://cp.blacknight.com > Email > Email Addresses
# Click on the affected service user name in the list
# Go to Email Addresses > Add > Add the email address that no longer works > Submit

That should start to work again straight away.  If you continue to have problems, or have any questions, please let us know as soon as possible.

When: Monday August 16th from 02:00 until 8am

What: Control panel software, provisioning system, Agents on all hardware nodes, mysql nodes, web servers will all be upgraded from version 2.9.4 to version 5.0. During this window access to the control panel will be restricted. However e-mail, hosting etc services should not be affected by this upgrade.

Changes made to webspaces, new or existing e-mail accounts added or modified, new database creations etc during this window would be ill advised.

Services Affected:

cp.blacknight.com - Control Panel only.

Windows Shared hosting servers may be restarted. only if the upgrade of the management application proves to be problematic.

Linux Shared hosting servers may be restarted. only if the upgrade of the management application proves to be problematic.

Exchange Servers may be restarted, but due to the design of the Hosted Exchange no downtime should be noticed.

Qmail servers will have their imap/pop3/smtp services restarted as new versions of the software get put in place. Also changes may be made to LDAP so there could be intermittent authentication issues.

Sitebuilder servers should not be affected.

VPS nodes will have some software upgraded, but end users should experience no downtime.

Domain registrations and modifications will not be affected by this upgrade.

Update 07:40 Monday 16th:

The upgrade is still being performed at the moment, but it's in it's final stages. We'll post an "all clear" notice when it's complete.

Update 08:45 Monday 16th:

During the upgrade Parallels appear to have broken the provisioning system somehow. As this drives the Control Panel the CP is still down. They're working to resolve this issue but we've got no ETA as of yet.

Update 1045
The maintenance and upgrade has been completed. If anyone has any specific issues post-upgrade please contact our helpdesk

Our engineers are currently seeing issues with one of our older Direct Admin servers:

Ector.blacknight.ie - 81.17.252.50

The server was rebooted not long ago and there are some lingering issues that may affect the receiving of email and possibly cause downtime on your websites.  We are working on this issue at the moment and our highest priority is to get this server back up and running as soon as possible.

We also hope to implement some changes on this server over the upcoming week to resolve the intermittant issues this server has been having over the last couple of weeks.


Update 12:50 - The server is back up now and our engineers are checking all services to ensure they are up and running.  They are still working on the server to restore all back to a normal level of service.

Update 15:10 - I'm afraid the server has gone down again and our engineers are working on restoring service once more.  We are still investigating the root cause of these issues today.

Update 15:26 - The server is still a little slow but services are back now.  We will be contacting some of the busier sites on this server in order to alleviate the load issues.

Update Aug 5th 11:22 - This server has had some config changes, we've moved a few of the busier wordpress blog sites off it and also we've found a problem in the AntiSpam system that was causing lookups on dead realtime blacklists. All of these changes appear to have resolved the problems we were seeing on this server for the past couple of weeks. We'll be monitoring it closely for the next week and if there are no issues in that timeframe we'll consider this issue closed once and for all.

Summary: We're going to tweak the storage backend for mail.blacknight.com tonight around 22:00 hours. We estimate approx. 30 minutes where e-mail, webmail, imap(s), pop(s) and smtp will be unavailable.

When: Tuesday August 3rd at 22:00 hours until 22:30

What: We've had some complaints about sending and receiving being slow, this is caused by a problem in Courier-Imap where it does a "list" on the users home dir during the authentication process before a list is requested from the mail client. We believe that the NFS servers have more capacity so we wish to restart the daemons after some configuration changes. As this will mean the nfs shares will have to all be unmounted and offline we'll need to shutdown all mail services while we do this. The change isn't major and should take less than 30 seconds, however we have to ensure that no mail is lost so we're shutting the system down.

We also have further storage nodes to go into the cluster but we're putting off this install until we stabilise the current solution.

Our technical team are currently working to resolve an issue with the mail cluster.

This issue is impacting some people's ability to login.

More details once we have them

UPDATE 1045: the issue appears to be related to LDAP

UPDATE 1052: We have a ticket open with software vendors. Our technical team continues to work on the issue as well

UPDATE 1107: The technical team continue to work on the issue. Some people are reporting continued issues, while others appear to have full service

UPDATE 1130: We've made some adjustments to the mail cluster configuration. The technical team are still working on it

UPDATE 1154: The mail cluster is not stable. We are working on it. Once we are confident that the issue has been resolved we will update this blog post

We understand that people are frustrated with the mail issues this morning, but we are working on the issue and have been since we first became aware of it.

We do not know when the issue will be resolved as yet.

UPDATE 1210: In response to some of the queries and comments people have posted. The issue is specifically related to "authentication" ie. logging in to mail to send / receive. Mails from the outside to you should not be affected.

Unfortunately with the number of people trying to collect and send emails simultaneously the servers are under high load so service may be slower than normal

UPDATE 1400: Email service is currently stable. If you are still having issues please contact our help desk

We are currently experiencing some issues on our newer shared mailservers and this will affect any customers that have email through a shared hosting package set up on http://cp.blacknight.com - though not our Hosted Exchange customers.

The issue is only with logging in to access your emails so any emails sent to your address during this time will be sent on to your mailbox and be ready for you to download as soon as this issue is resolved.

Currently our engineering are working on this problem and it is related to the service that handles the login and authentication of users accessing their email.

UPDATE 1138
Service is currently degraded, but functional. We are still working on the situation.

UPDATE 1155
Service has been restored, however we will be monitoring it closely

Summary: mail.blacknight.com and it's associated services are running slower than normal today due to increased activity from end users. This increase in activity is not part of the normal pattern of use so we're trying to pinpoint the cause.

Services affected: smtp, imap and pop3.

ETA For a fix: Approx 1 hour

Update: Jan 18th @ 12:41 pm

We've switched storage appliances on the back end of the mail cluster. E-mail is out of sync by 1 day but it's currently synchronising. As a result of this mail auth, and general mail delivery, connection times have decreased dramatically.

You may get some duplicated e-mail, but this was preferable over e-mail being down for several hours while we fight with the current storage platform.

Our technical team is currently working on an issue impacting our mail cluster.

The directory services are having some issues, so authentication is currently not working

In simple terms - people cannot login to collect or send email

If you need to send email urgently please change your email SMTP to that of your ISP (broadband provider)

We will update you as soon as we know more.

Services affected: webmail, pop, smtp, IMAP

UPDATE 2137
We have been running the cluster in a degraded state for the last 45 minutes approximately and are monitoring the situation.

UPDATE December 2nd 1426
Apologies for not updating earlier
We have been monitoring this situation closely and the cluster appears to be handling authentication without errors. If you do, however, have any issues please let support know.

We are currently experiencing some intermittent issues with our mail cluster located at mail.blacknight.com

Our engineers are currently working to resolve this issue asap.

We will update this blog post once operations are back to normal.

UPDATE 16:16: The erroneous server has been removed from the mail cluster completely. This means all services are back to normal but missing some power. We are working to get the server back online.

UPDATE 16:44: The server has no been brought online and introduced back into the cluster.

UPDATE 17:06: Due to inconsistencies with ldap, we are trying to rebuild the index. This is causing some issues currently which we are working on now. Please bare with us.

UPDATE 17:10: Mail is being routed correctly now. We are monitoring it closely.

UPDATE Nov 25th 11:30: There's still some lingering issues with the mail cluster which are causing POP timeouts. We're currently investigating and working with our vendors in order to try and narrow down exactly where the problem is occuring.

UPDATE Nov 25th 12:40: This issue is on going. There appears to be a locking issue on the nfs server which is being caused by a huge volume of pop3 connections coming into the servers.

UPDATE Nov 25th 15:00: This issue is still on going. We're aware 1000s of pop3 and imap users can not currently access e-mail. The problem is still be worked on and we hope to have services restored today. We may need to put an emergency maintenance window in place in order to fully diagnose the issue. Right now the engineering team is tracing the high load on the mail servers back to the NFS file server and the storage it uses.

The storage is provided by an EMC Clariion CX300 Fibre Channel SAN. This should facilitate 100s of thousands of concurrent users however it seems that at extremely busy times it can't cope. The main reason for this appears to be the fact that the pop3 process immediately does a read on Maildir folder for the mailbox that is logging in without the list command being sent to by the client application. That means that every single auth command creates an IO intensive job that the mail servers have to deal with. We're doing everything within our power to resolve this and we've put a new storage array into the data centre. We hope to bring this online tonight. During which time we'll have to take mail completely offline.

Points of Interest:

1) Both pop3 and imap are affected by this issue.
2) Delivery of inbound e-mail to your inbox is unaffected.
3) if we do decide on a maintenance window, we'll store all e-mail off the cluster and forward it on once the qmail cluster has returned to good health.
4) During the move to the new system, your old e-mail may dissappear, however it'll arrive back in your imap folder within 24-48 hours as we sync data back from the old storage server.
5) we're working with Parallels to devise a far better storage arrangement and configuration of the qmail cluster. This has been on going for a couple of months but we believe there is an end in sight.
6) Our support team are trying to deal with all your calls, e-mails and live chats as quickly as possible. Please bear in mind that the problem is out of their control and that the most up to date information on the issue will be posted here on the status blog.

Services affected currently: Webmail, imap and pop mail - smtp should continue as normal.

UPDATE Nov 26th 09:04: Services are resumed but at a degraded rate currently. Our engineers are currently executing a plan of action to get the system back up and running at full speed again. Please bare in mind the sheer volume of data we are working with, this is the primary factor on why this issue is taking some time to resolve.

Our Shared Hosting Qmail cluster is currently experiencing some problems. This has resulted in some people experiencing timeouts collecting their emails.

 

More on this as we find out more.