Results tagged “Emergency maintenance” from Blacknight Hosting Network Status

Our main site is currently not accessible due to a network issue

The store, control panel, support desk and all services related to hosting, domains, dns etc., are NOT affected

We are currently working on resolving the issue as quickly as possible

UPDATE 2208 - issue has been resolved

Summary: Today at 16:39 customers on several firewall segments were impacted by high packetloss. This was the result of a yet unknown event, however we were able to restore service within a timely manner by forcing the "active" firewall into standby thus forcing the standby firewall to take over.

We're currently investigating the cause further and examining syslogs, port graphs, cpu graphs etc. We will post a conclusion to the issue once our engineering team have completed their investigations.

Current situation as of 17:10:

Network connectivity was restored within 10 minutes of the actual event occuring however some services like e-mail on our shared hosting cluster will take some time to recover as there are an extraordinary number of inbound connections into the cluster and the load balancer is doing it's best to diviy out connections to the mail servers.

Update 1810

Mail services should be back to normal, but if there are any issues let us know.

Currently our primary control panel is throwing errors for new logins. We're working on this issue and we've escalated it to our control panel vendors.

Current ETA is unknown at this time.

Update: 16:05

The control panel has been backup since 16:05. The tomcat process that drives the UI for the control panel has been having issues of late and we're awaiting a fix from Parallels to get rid of the problem. In the mean time we're monitoring this very closely and we'll post further updates here as we have them.

Summary: Windows node pemvzwin02 is experiencing issues with internal virtuzzo API calls. This is causing a task backlog in our provisioning system. In order to diagnose the issue further we've been asked to reboot the node.

We're taking the opertunity to run windows updates while we do this, so at least some good will come of it.

The following list of VPSs are affected:

vps-238.cp.blacknight.com, vps-258.cp.blacknight.com, vps-262.cp.blacknight.com, vps-265.cp.blacknight.com, vps-271.blacknight.com, vps-282.cp.blacknight.com, vps-287.cp.blacknight.com, vps-288.cp.blacknight.com, vps-302.cp.blacknight.com, vps-317.cp.blacknight.com, vps-331.cp.blacknight.com, vps-337.cp.blacknight.com, vps-341.cp.blacknight.com, vps-342.cp.blacknight.com, vps-343.cp.blacknight.com, vps-344.cp.blacknight.com, vps-346.cp.blacknight.com, vps-347.blacknight.com, vps-353.cp.blacknight.com, vps-354.cp.blacknight.com, vps-355.cp.blacknight.com, vps-357.cp.blacknight.com, vps-361.blacknight.com, vps-362.cp.blacknight.com, vps-386.cp.blacknight.com, vps-387.cp.blacknight.com, vps-390.cp.blacknight.com, vps-407.cp.blacknight.com, vps-409.cp.blacknight.com, vps-410.cp.blacknight.com,
vps-584.cp.blacknight.com, vps-589.cp.blacknight.com, vps-612.cp.blacknight.com, vps-620.cp.blacknight.com, vps-626.cp.blacknight.com, vps-635.cp.blacknight.com, vps-642.cp.blacknight.com, vps-645.cp.blacknight.com, vps-664.cp.blacknight.com, vps-667.cp.blacknight.com, vps-672.cp.blacknight.com, vps-684.cp.blacknight.com, vps-689.cp.blacknight.com, vps-704.cp.blacknight.com, vps-709.cp.blacknight.com,
vps-710.cp.blacknight.com, vps-716.cp.blacknight.com, vps-718.cp.blacknight.com, test.zerotouch.ie

I'll update this ticket once it comes back up and all VPSs are back up. It'll probably take upto an hour, but it's completely necessary.

Update: 21:30

The node rebooted successfully and about 70% of the VPSs have restarted. The rest should be back online in the next 10 to 15 minutes or so.

We're continuing our investigation of the issues we've been encountering in our backend systems.

Summary: Mail on our new shared hosting platform is not working 100%, there was a change made by Parallels to our control panel systems this morning which should not have affected e-mail but it looks to have had.

We're currently working to resolve this issue and we'll post updates when we have an ETA.

Update: 09:55

This appears to be an LDAP problem, we're working to resolve it at the moment. Right now the service is running but keeps locking up. The knock on effect is that the mail servers can't auth pop3 or imap users currently.

Update: 10:35

We've been monitoring ldap now for some time and for the last 18 or so minutes we've been successfully authing users. We have the capability for LDAP replicas which we'll implement to prevent us having to rely on 1 LDAP server in the future.

Problem resolution:

The ldap indexes were corrupt, we stopped ldap, moved the indexes out, started ldap and the indexes have been rebuilt. POP, IMAP and SMTP are now functioning correctly. 

The shared server Rivalin is currently experiencing issues.

Our technical team is working on a resolution and we hope to restore all services shortly

Edit: Paul Kelly, our CTO, has added more detail - looks like we were both posting about the issue at practically the same time!

We're apply window and Virtuozzo updates this morning to some of our windows VPS hardware nodes.

It'll take approx 20 minutes for each one to complete reboot and come back up.