mail.blacknight.com / smtp1r.cp.blacknight.com emergency maintenance

Message

Summary: This evening we're going to go ahead and do the work we said that we'd do in http://blacknig.ht/18d .

We'll start at 19:00 and it shouldn't take longer than an hour or so.

Update 20:00: The mail is syncing between the old and new storage. It's about 50% done. We suspect it won't take more than another hour or so.

Update 20:19: FYI this affects all Qmail services, so pop/imap and smtp. Including pop33r.cp.blacknight.com etc

Update 20:45: This isn't quite complete yet. It might run past 21:00 GMT and if it does we'll roll back the changes and try again later tonight. We'll leave the sync running while mail continues to flow and put pop3/imap services back live.

Update 21:00: We've backed out of the move for now. We thought that the sync of 24 hours of data would only take 45-60 minutes, unfortunately it's still going. We'll go again in a few hours when less people are waiting on e-mail.

Update Saturday 5th @ 00:05: The data sync has been on going for the past few hours and is almost complete. At this stage we're going to block inbound SMTP e-mail and leave pop3/imap alive until we're ready for the final move to the new platform. By disabling SMTP inbound we won't have to re-sync from scratch again all the new e-mail that arrives.

Update 02:20: The data sync is complete. People find the odd e-mail from earlier this evening marked as unread or it'll download again via pop3. This was unavoidable. however the good news is that we've moved over to the new storage platform and so far it's performing far better, however this is a quiet time for the cluster.

Webmail/pop/imap/smtp access has now been fully restored.