December 2009 Archives

We are currently experiencing some mail issues locally. Some users may receive bounce-backs when submitting a ticket through our helpdesk or email staff directly.

Our engineers are working on this currently and details will be posted on this blog once normal service is resumed.

In the meantime, our telephone lines are opened as per normal should you wish to reach us.

For clarity sake this was an internal issue. Our office exchange servers had some issues after the migration of Active Directory services from a legacy server to a new server. We also moved DHCP services internally in our office to new servers last night and both these issues caused our internal mail problems.

Update: 09:40 Problem fixed: Exchange is 100% back now and mail is being delivered to us.

This server is currently experiencing difficulties. We hope to have a resolution as soon as possible.

14:30 - all issues appear to have been resolved.

15:15 - It appears that there is still a problem with .Net Applications on this server. We are looking into it.

18:30 - All websites hosted on this server are now back to normal. We apologise for any inconvenience caused. If your site is sill having problems, then please email support@blacknight.com

Our cp interface is currently having some issues.

We're currently working with our software vendors to try and get it back up and running.

All services are currently up and unaffected, only client access to their control panel is down at the moment.

11:00 - the Control Panel has been up and running without error since 04:00 this morning.

We are currently experiencing issues with our Windows VPS Hosting hardware node, PEMVZWIN06.

An engineer is currently on route to the data center to resolve this as soon as possible.

We will update this post with the causing issue and status updates once they become available.

7:25AM : The issue has be located to be a power supply failure on the machine. Engineers are working on this currently.

7:44AM : The PSU has been replaced and the server is now coming back online. VPSs should be available within the next 10 to 15 minutes.


This is now closed.

Due to the increase amount of traffic to the shared linux servers located on the hardware node, PEMVZMPS19, we will be increasing it's performance potential with a CPU upgrade.

The affected services are:

pemlinweb21.blacknight.com - 81.17.254.58
pemlinweb22.blacknight.com - 81.17.254.59

The time frame for the upgrade will commence at 22:00 on the 9th of December 2009 and should last no longer than 15 minutes of downtime.

We will update this blog post once completed.


Update 10:12 This has been completed.

The shared hosting linux server, PEMLINWEB12(81.17.254.94) needs to be brought for an urgent reboot.

At present the server is registering only a fraction of it's physical cores causing the load to become quite high.

The estimated downtime is less than 5 minutes but needs to occur asap to ensure all clients receive the maximum from the server.

I will update this blog post once completed.

Update 11:09: This is now complete

Update 11:17: One last reboot required, ETA 3 minutes.

Update 11:20: Server is now complete. Thank you.

We are currently experiencing some issues with our shared hosting node PEMLINWEB12 (81.17.254.94).

Our engineers are working on this issue currently and will update his blog post with more information as it becomes available.

Update: 10:30PM This issue has now been resolved.

A CPU fan has failed in the above server. We are replacing it right away, and the server has been shut down as a precaution. We expect it to be back up and running again in the next 20 minutes.

Update: 16:40 December 3rd

We've been monitoring this machine since the heatsink swap earlier today however it's still experiencing higher than normal thermal output. We're going to take it offline just after 17:00 today to investigate the issue. It'll be down for approx 15 minutes while we investigate the problems. It should return back into service fully by 18:00 today. This is providing we don't have to do much other than re-sit the heatsink onto the CPU. If further work is required we'll post here with updates.

Update: 17:28 December 3rd

The issue was found to be a hardware failure. This has now been replaced with new equipment and the CPU core is operating at it's normal temperature. Of course we'll be monitoring this over the course of the night. All VPSs are back online also.

Summary: Today this server has powered itself off twice. We've been working on the issue since this morning at 8am but it appears that it may be due to a cooling problem localised to this rack.

We may have to swap it's position in the rack and while doing so we'll check that all the fans are operating properly, if they're not we'll swap them.

We've ordered additional vented tiles for this part of the data centre also which will improve air flow for the servers in this rack. These will be installed today.

The following VPS' are affected by this issue:

vps-642.cp.blacknight.com
vps-841.cp.blacknight.com
vps-890.cp.blacknight.com
vps-1110.cp.blacknight.com
vps-1118.cp.blacknight.com
vps-1126.cp.blacknight.com
vps-1142.cp.blacknight.com
vps-1164.cp.blacknight.com
vps-1194.cp.blacknight.com
vps-1198.cp.blacknight.com

We hope to have service reliably restored within the next few hours. An emergency maintenance window may be necessary.

Update: 11:31 December 3rd

We're shutting this machine down to replace a dead fan on CPU0 which is causing CPU0's core temperature to rise in excess of 100C and the machine to shut itself down in order to protect the CPU.

Update: 11:42 December 3rd

This machine is back in production now with a new heatsink. VM's are starting up and should be fully online by 11:55 or 12:00.

Update: 11:56 December 3rd

All VMs are now back. The core temperatures of all CPUs in this node are now returning to normal. This issue is now closed.

When: 15:30 on December 2nd 2009
Summary: One of the IP RBL's we check all monitored IPs against was returning "true" or "valid" results for all lookups today. This is because the domain in question that the IP blacklist uses expired.

As a result all customers with server monitoring via sms and e-mail will have gotten an alert with the subject of:

IP Blacklist Report For: $servername

In the body of the e-mail it contains information about the blocking list. This was sent entirely in error on our part. it was however our of our control. To prevent further events re-occurring we've disabled that particular blacklist and trimmed the others that we check against as well.

Our technical team is currently working on an issue impacting our mail cluster.

The directory services are having some issues, so authentication is currently not working

In simple terms - people cannot login to collect or send email

If you need to send email urgently please change your email SMTP to that of your ISP (broadband provider)

We will update you as soon as we know more.

Services affected: webmail, pop, smtp, IMAP

UPDATE 2137
We have been running the cluster in a degraded state for the last 45 minutes approximately and are monitoring the situation.

UPDATE December 2nd 1426
Apologies for not updating earlier
We have been monitoring this situation closely and the cluster appears to be handling authentication without errors. If you do, however, have any issues please let support know.