Results tagged “Qmail” from Blacknight Hosting Network Status

When: Monday August 16th from 02:00 until 8am

What: Control panel software, provisioning system, Agents on all hardware nodes, mysql nodes, web servers will all be upgraded from version 2.9.4 to version 5.0. During this window access to the control panel will be restricted. However e-mail, hosting etc services should not be affected by this upgrade.

Changes made to webspaces, new or existing e-mail accounts added or modified, new database creations etc during this window would be ill advised.

Services Affected:

cp.blacknight.com - Control Panel only.

Windows Shared hosting servers may be restarted. only if the upgrade of the management application proves to be problematic.

Linux Shared hosting servers may be restarted. only if the upgrade of the management application proves to be problematic.

Exchange Servers may be restarted, but due to the design of the Hosted Exchange no downtime should be noticed.

Qmail servers will have their imap/pop3/smtp services restarted as new versions of the software get put in place. Also changes may be made to LDAP so there could be intermittent authentication issues.

Sitebuilder servers should not be affected.

VPS nodes will have some software upgraded, but end users should experience no downtime.

Domain registrations and modifications will not be affected by this upgrade.

Update 07:40 Monday 16th:

The upgrade is still being performed at the moment, but it's in it's final stages. We'll post an "all clear" notice when it's complete.

Update 08:45 Monday 16th:

During the upgrade Parallels appear to have broken the provisioning system somehow. As this drives the Control Panel the CP is still down. They're working to resolve this issue but we've got no ETA as of yet.

Update 1045
The maintenance and upgrade has been completed. If anyone has any specific issues post-upgrade please contact our helpdesk

Summary: Most heavy users on our system have been experiencing some slowdown of e-mail for the last couple of weeks. This was up until recently intermittent however it is now a problem that occurs during peak usage times. i.e. between 11am and 5pm or so.

The cause for this issue appears to be because of a number of factors, some of which are out of our control.

Within our control our storage array that is well speced doesn't appear to be doing the business. i.e. under heavy slow it's quite slow to respond. We've taken this up with the vendor in question, however we're building a second SAN to replace it. We hope to put the new storage in place during the weekend of October 17 and 18th. This should give much better performance over all and will also stop the mail delays.


Outside of our control we have ldap and whosond which are services parallels use to maintain certain databases of user information that qmail and couier-imap use to get information about users. Each time these have issues it causes problems with mail delivery, this compounded by the storage array being slow is the cause of the problems.

We've also got ram upgrades which we're going to put in place tonight on all the mail nodes,
because we're clustering them there'll be minimal impact during the ram upgrades.

Please bare with us while we get this issue resolved.

Our Qmail cluster was experiencing some issues earlier this morning.

For an unknown reason which is still being  investigated, qmail stopped closing connections correctly and eventually ran out of available connections causing the outages.

Forcefully killing off the stuck qmail processes and restarting qmail fixed the issues, and it's still being monitored

UPDATE: 12:24

On further investigation our technical team found some other issues, which would have impacted end users

In essence LDAP had issues. As LDAP looks after all authentication then other services would have appeared to be degraded.

We're investigating an issue with our qmail cluster at the moment. This issue is causing auth problems for pop and imap however mail is being received into the system so it's not being lost.

I'll post an update soon as we've got further information.

Update: 11:15

This issue is now resolved. The main issue was IMAP auth, further testing showed that POP3 was working but IMAP wasn't. We've restarted services and rebooted each mail server in succession and this has resolved this issue.