mail.blacknight.com / smtpr1.cp.blacknight.com mail delivery delays

Message

Summary: All domains serviced by the above mail server cluster are experiencing a mail delivery delay to inbound e-mail. Mail being delivered to external addresses via the same system is not affected by the issue.

We suspect it's a problem with the spam-assassin servers and we're checking the configurations to see can we discover the source of the problem. We'll post further updates as we have them.

Update: 12:50: We've found the source of the problem. We are experiencing some internal packet loss between the SA servers and our resolvers in their local data centre. We should be able to fix this issue in the next 30 minutes and then the mail servers should be able to catch up.

Update: 13:50: The mail queues have reached their peak and they're now starting to go down quite quickly. The issue was indeed the cause of a DNS issue caused by packet loss on our SA nodes. The packet loss was due to some limits that were put in place on a number of visualised servers, we've raised these limits and we're seeing much higher throughput now. We expect the queues to have returned to normal by 15:00 approx. We'll post one more update closer to 15:00.

Update: 15:00: Unfortunately the issue we found earlier hasn't completely fixed the problems that the mail cluster is experiencing. We've asked our software vendor to have a closer look at the configuration for us as it hasn't really changed since May. They provide all the components involved so they might be able to assist us further.

Update: 16:00: The mail queues are going down slowly however the cluster hasn't completely stabilised as of yet. We may require an outage window of about 1 hour tonight from 23:00 to 00:00 in order to fully rectify the situation.

Update: 17:00: Right now Parallels have made some changes to the concurrency of the delivery daemon on the mail servers. The default setting appears quite low so this is being put up which should begin delivering email quicker. We still have a route cause of the problem and we'll continue working on this to get to a resolution. The next update will be at 20:00.