Recently in email Category

Summary: Nothing. After the downtime on Monday and Tuesday we put a new Mail system in place that deals with POP and IMAP. it's more robust, faster for you and for us. It's better in many different ways than the old one.
There is a slight drawback -  you can no longer send e-mail without explicitly turning on SMTP authentication in Outlook, Apple Mail etc.

We have gathered a few articles for you to help you out on this:

1) https://support.blacknight.ie/index.php?_m=knowledgebase&_a=viewarticle&kbarticleid=472
    - this gives you instructions on how to enable this in most common e-mail clients.

OR

2) http://wiki.blacknight.com/index.php/SMTP
    - this gives you a list of SMTP servers for most Irish ISPs, down the bottom is a link to a page that has a list of ISPs for many ISPs outside of Ireland.
An ISP is the company that provides your broadband, cable, dsl, dial up, satellite net connect etc.

I've requested that a newsletter go out to all customers with the above information in it as a technical notice for the next few runs.

Update: 09:35

Webmail users using Firefox 4 or IE 7 or above can use our "alternative" webmail application here: https://altmail.blacknight.com

Summary: We have had some complaints about mail delivery last night that it was delayed. We have found that the cause of this is similar to an issue we've found on other machines with 10GigE interfaces that are very new. There is a new driver that we will apply to all machines involved and that will prevent the issue from occurring again.

This affected inbound e-mail into our spam scanning server that sits in the Cloud. No e-mail was lost during this window. It was resolved around 00:00 last night.

Summary: At 23:00 tonight Friday 21st of October we're going to update the NIC card drivers for the 10GigE network cards that provide connectivity to the SAN. To do this we have to take mail down for a window of approx 15 minutes.

This is to hopefully resolve a bug that we've seen manifest a number of times since the mail storage platform was upgraded last Friday night. Once this is complete we hope that the mail system will function better moving forward.

Summary: 2 of the 4 servers that deliver mail to mailboxes currently have a few thousand messages in their queues at the moment. We expect this to die down in the coming 60-90 minutes as it was due to a number of large mailers from various companies hosted outside the qmail cluster. As a result the spam scanning servers are very busy.

Additionally we're moving the primary spam scanning box to new hardware which should improve performance 10 fold and increase delivery times in periods of high inbound email load.

Update:
The upgrade has greatly improved performance.

We're currently experiencing issues with our qmail cluster, mail.blacknight.com - Our engineers are working to resolve this ASAP and all updates will be posted here as they become available.

Summary: Following on from this weeks earlier maintenance we're going to do the final move of e-mail to the new storage platform tonight October 14th.

We'll take mail offline again at 23:00 hours for approx 2 hours until 1am. This move is essentially just an rsync of data from the old system to the new system and that's it. We've already seeded the data to the new platform using a recent backup so there isn't a huge amount of data to be copied. We will then re-mount the new drives onto the mail servers and begin the mail flow once again after some essential tests.

Tomorrow Saturday 15th we'll monitor everything closely and see how it all performs.

Summary: Since approx 11:30 this morning people have been experiencing some delays in downloading e-mail, it appears to download very slow and sometimes it times out. Between 11:30 and approx 13:30 is the busiest time of day for the mail servers and we expect that a rebuilding raid array may be causing this issue. The array should be rebuilt completely by mid to late afternoon today at which time things should improve dramatically. We can't really do anything to speed this up in the interim.

Additionally once we perform the major upgrade later this week of the storage backend we expect mail to function 100% and perform much much better than it has been.

Update @ 16:45:  Mail raid rebuild is still continuing at this time.

Summary: On 23:00 on Monday 10th of October we're going to take the mail cluster offline completely for approx 4 hours until 03:00. This is to facilitate the following:

1) Upgraded storage space for the mail store
2) Upgraded performance for the mail store
3) Upgraded file system from ext3 to ext4 for performance reasons

We're doing this in stages.

Stage 1)

We've restored the existing mail system from a recent backup to a new location. We'll begin copying that data to the new Mail Storage node which has 20Gbit/s of connectivity to our SAN. We estimate that this will take approx 24-48 hours to complete.

Stage 2)

We will then take the mail system offline again in a couple of nights to do a final sync of the changes on the new system. This will depend largely on the amount of time it takes to do the copy over to the SAN. Estimations range from between 36 and 50 hours approx.

Tonight we're simply relocating the servers to a new rack in close proximity to our SAN. And while they're down we're doing a clean up of old mailboxes and purging data that is no longer required.

Summary: We're noticing some strange behaviour in the mail system right now. For no reason at all the load balancer is not putting connections through to the mail server on port 25. We're currently investigating this and hope to fix it shortly.

Additionally there is around 6k or so mails in the "local" delivery queue. Which means that inbound e-mail delivery is slower than normal. Again we're investigating this issue but it may be related to the issue mentioned above.

We will post further updates on this when the information is available to us.

Summary: In order to get the system stable we're taking all services offline for about 1 hour to do some configuration that we hope will speed things up.