August 2011 Archives

We're currently experiencing a network issue which appears to be affecting large parts of our shared / VPS network.

Our technical team are aware of the issue and are working on it

We will update with more information as soon as we have it

UPDATE 2300 - the issue appears to be a DDOS against our DNS servers which is affecting any service that uses them.

UPDATE 0005 - Our network engineers are working on the issue. Due to the nature of the attack it is impacting sites and services on our shared network ie. more than DNS.

UPDATE 0017 - our main site, control panel and most client sites *should* be accessible now. We're confirming any outstanding issues

UPDATE 1135 - The network issue was fully resolved last night. One or two clients reported issues this morning that were unrelated.

With respect to the issues last night I am still waiting on a more detailed explanation from our technical team, but in simple terms what happened was as follows

Something / someone launched a large attack against us sending a very large amount of DNS queries / packets of data. The huge spike in queries basically maxed out the firewalls that protect that segment of our shared hosting network.

Once our technical team were able to identify the network block that was being hit they were able to take actions to remedy the situation.

All shared hosting services were functional again from around midnight last night. We are aware of one Windows webserver that had an issue last night, but it is, as far as I know, unrelated.

If we can get a more detailed explanation of last night's issue we'll post it here / edit the post

We've had to reboot the hardware node containing these two pemlinwebs. It should hopefully be back up in 10-15 minutes.

UPDATE 18:15: Both server are back up and running.

pemvzwin15 will be offline Wednesday the 31st August for approximately 50 minutes from 6pm to allow for updates and reboots.

Due to an excessive swapping issue we are seeing, we want to apply some patches to a hardware node. 

One of our older Direct Admin based shared hosting servers is currently experiencing a backlog of email in its mailqueue and this would result in a delay in emails coming through to addresses hosted on this server.

The server in question is Igraine.blacknight.com - 81.17.252.25

Please note this does not affect any other shared hosting server, or email accounts hosted through cp.blacknight.com or our Hosted Exchange service.

We had to reboot igraine as it became unresponsive and we're just waiting for the boot up to finish.

Update 13:47: This server is backup, and we're working on fixing the root cause of the problem.

pemvzwin15 will be offline Thursday the 25th August for approximately 40 minutes from 9pm to allow for updates and reboots.

We apologise for any inconvenience

Update 21:45:

Server remains offline, working on getting it back online as soon as possible

Affected VPS':

vps-1996.cp.blacknight.com

vps-2152.cp.blacknight.com

vps-2274.cp.blacknight.com

vps-2276.cp.blacknight.com

vps-2304.cp.blacknight.com

vps-2308.cp.blacknight.com

vps-2313.cp.blacknight.com

vps-2315.cp.blacknight.com

vps-2317.cp.blacknight.com

vps-2325.cp.blacknight.com

vps-2328.cp.blacknight.com

vps-2330.cp.blacknight.com

vps-2332.cp.blacknight.com

vps-2334.cp.blacknight.com

vps-2341.cp.blacknight.com

vps-2342.cp.blacknight.com

vps-2345.cp.blacknight.com

vps-2348.cp.blacknight.com

vps-2351.cp.blacknight.com

vps-2354.cp.blacknight.com

vps-2366.cp.blacknight.com

vps-2367.cp.blacknight.com

vps-2384.cp.blacknight.com

vps-2415.cp.blacknight.com

vps-2416.cp.blacknight.com

vps-2417.cp.blacknight.com

vps-2420.cp.blacknight.com

vps-2441.cp.blacknight.com

Update 22:20:

Server back online now

Summary: The Servers listed below require a reboot to faciliate maintenance and upgrades. The Servers will be offline Wednesday the 24th of August between 6pm and 8pm for approximately 20 minutes.

pemmysql05.blacknight.com      81.17.254.60

pemlinweb25.blacknight.com     81.17.254.64
pemlinweb26.blacknight.com     81.17.254.67

Morgana

Rivalin is currently unresponsive. We've got an engineer on site looking into it.

There is an issue with the control panel system and Windows subscriptions at the moment.  We have raised it with the control panel vendors and hope to have it resolved as soon as possible.

Please note that this should not affect your websites or email access at all, it is just the communication between the control panel and windows subscriptions specifically.  This may prevent you from logging in to cp.blacknight.com if you only have a Windows subscription (or a Windows subscription is the default subscription on your account), or from selecting a windows plan under Select Subscription if you have multiple subscriptions.


Update - 15:06: I'm afraid this issue is not yet resolved but the entire pem service needed to be restarted to help resolve it.  This would have brought down the cp.blacknight.com login screen itself for all users.  We hope to have this back up shortly.

Update - 15:15: This issue should be fully resolved now.

Eurid, the .eu registry, is conducting scheduled maintenance tomorrow morning from 0600 to 0900 CEST.

During this window WHOIS, new registrations and updates to .eu domain names will be interrupted.

Existing .eu domain names will not be affected.

UPDATE 0900 - Eurid inform us that the maintenance is now complete and all services should be back to normal

Summary: The above named node is currently experiencing high load. Our engineers are investigating this and hope to have it resolved shortly. The mysql service had also maxed out it's connection pool so people may have seen errors about this.

Update 08:38: this is now resolved, we're monitoring this node closely.

The above named Linux Shared Hosting node, IP address 78.153.214.51, will be offline for a period of approximately 90 minutes from 22:00 this evening.

All sites hosted on this node will be unavailable for this time period

Update 22:38 - this node is back online, and all sites hosted on it have returned to normal.

I'm afraid we are currently experiencing a very high load on one of our VPS nodes this morning.  Our engineers are working on this now and we hope to have it resolved as soon as possible.

The main IP addresses of the affected VPS plans are:

78.153.210.173
78.153.210.74
78.153.209.45
78.153.209.65
78.153.209.100
78.153.209.128
78.153.209.109
78.153.209.139
78.153.209.178
78.153.209.179
78.153.209.194
78.153.209.196
78.153.209.198
78.153.209.231
78.153.210.12
78.153.210.37
78.153.210.48
78.153.208.98
78.153.210.63
78.153.210.71
78.153.210.104
78.153.210.119
78.153.210.202
78.153.210.218
78.153.210.236
78.153.210.249
78.153.211.43
78.153.211.91
78.153.211.119
78.153.211.164
78.153.211.167
78.153.211.168
78.153.211.171
78.153.211.184
78.153.211.89
78.153.211.188

UPDATE 10:20: We've had to reboot the hardware node. It's already on the way back up, and we're just waiting on the VPS to start up.

The above named Linux Shared Hosting node, IP address 81.17.254.59, will be offline for a period of approximately 90 minutes from 22:00 this evening.

 

All sites hosted on this node will be unavailable for this time period.

Update 22:35 - this maintenance window is now closed, and all sites are back online.

Summary: The above named Windows VPS node is being hard rebooted because it has become unresponsive. We estimate around 15 minutes for all the VPS' to come back online.

The following list of Containers are affected:

78.153.196.92
78.153.196.87
78.153.196.101
78.153.196.102
78.153.196.075
78.153.196.103
78.153.196.104
78.153.196.105
78.153.196.114
78.153.196.115
78.153.196.116
78.153.196.117
78.153.196.124
78.153.196.191
78.153.196.194
78.153.196.195
78.153.196.197
78.153.196.199
78.153.196.216

PEMLINWEB54 is currently after going down and is refusing to come back up. Engineers are investigating at the moment.

UPDATE 15:20: The server is back up, and we're investigating why it went down in the first place.

Summary: We've noticed that a virtuozzo service on this node is acting strangely using a lot of memory so we're rebooting this node. This should improve performance for the containers on this node.

Summary: The above named node is being hard rebooted because it has become unresponsive. We estimate around 30 minutes for all the VPS' to come back online.

The following list of Containers are affected:

 78.153.196.92
 78.153.196.87
 78.153.196.101
 78.153.196.102
 78.153.196.075
 78.153.196.103
 78.153.196.104
 78.153.196.105
 78.153.196.114
 78.153.196.115
 78.153.196.116
 78.153.196.117
 78.153.196.124
 78.153.196.191
 78.153.196.194
 78.153.196.195
 78.153.196.197
 78.153.196.199
 78.153.196.216

Summary: The Servers listed below require a reboot to faciliate maintenance and upgrades. The Servers will be offline Tuesday the 2nd of August between 6pm and 8pm for approximately 20 minutes.

81.17.254.72
81.17.254.73

81.17.254.59

81.17.254.70
81.17.254.88

mysql359.cp.blacknight.com

81.17.254.64
81.17.254.67

81.17.254.38
81.17.254.39

78.153.214.62
78.153.215.155

78.153.215.164
78.153.215.165

Morgana

CADOR

Due to some swapping issues on a hardware node containing two mysql nodes we are going to need to bring it offline for a peroid of 15 minutes.