Blacknight Hosting Network Status

Shared Hosting server pwmvzwin06 has become unresponsive

2013-05-24T19:32:06Z

Shared Hosting server pwmvzwin06 has become unresponsive and we have dispatched an engineer to investigate : the following services   are affected

    78.153.209.166
    78.153.210.203
    78.153.210.81
    78.153.208.181
    78.153.209.135
    78.153.210.1

update: 21:20 the server has stabilised and all services are now resumed

Legacy Direct Admin server "Galahad" has been intermittently rebooting

2013-05-24T18:26:19Z

The Direct Admin server Galahad , has been intermittently rebooting and blue-screening. The server is running now , but we are migrating it to a virtual server as the hardware could fail. We have shut down the email on this server and will reinstate it once the virtual replica is running. We estimate this will take 3.5 - 4 hours.
Other services are still running for the moment.

We will update this post as soon as we know more

Update 00:15 The virtualisation had to be restarted because the server hardware has rebooted and is unstable. We are shutting down all services on this server until we have completed the virtualisation. We estimate another 4 -5 hours and will update the post as soon as we have more information.

Network Maintenance

2013-05-17T13:25:07Z

We will be upgrading a top of rack switch in one of our dedicated hosing racks (C01) on Saturday 25th May at 11:00. Customers in this rack will experience two brief outages as their connection is moved over.

Apologies for any inconvenience caused.

Update: this maintenance window has been deferred until 25th May.

Security updates : multiple nodes

2013-05-15T16:22:36Z

Approx 30 MPS shared hosting nodes will be rebooted between 6:45am and 7:15am tomorrow morning to patch critical security vulnerabilities. Nodes will be offline for approx 5-10 minutes each.

The following servers will be affected:

pemlinweb15.blacknight.com
pemlinweb16.blacknight.com
pemlinweb19.blacknight.com
pemlinweb20.blacknight.com
pemlinweb55.blacknight.com
pemlinweb67.blacknight.com
pemlinweb68.blacknight.com
pemlinweb73.blacknight.com
pemlinweb74.blacknight.com
pemlinweb99.blacknight.com
pemlinweb100.blacknight.com
pemlinweb101.blacknight.com
pemlinweb102.blacknight.com
pemlinweb103.blacknight.com
pemlinweb104.blacknight.com
pemlinweb105.blacknight.com
pemlinweb106.blacknight.com
pemlinweb107.blacknight.com
pemlinweb108.blacknight.com
pemlinweb109.blacknight.com
pemlinweb110.blacknight.com
pemlinweb111.blacknight.com
pemlinweb112.blacknight.com
pemlinweb113.blacknight.com
pemlinweb114.blacknight.com
pemlinweb115.blacknight.com
pemlinweb116.blacknight.com
pemlinweb117.blacknight.com
pemlinweb118.blacknight.com
pemlinweb119.blacknight.com
pemlinweb120.blacknight.com
pemlinweb121.blacknight.com
pemlinweb122.blacknight.com
pemlinweb123.blacknight.com
pemlinweb124.blacknight.com
pemlinweb125.blacknight.com
pemlinweb126.blacknight.com
pemlinweb127.blacknight.com
pemlinweb128.blacknight.com
pemlinweb129.blacknight.com
pemlinweb130.blacknight.com
pemlinweb131.blacknight.com
pemlinweb132.blacknight.com
pemlinweb133.blacknight.com
pemlinweb134.blacknight.com
pemlinweb135.blacknight.com
pemlinweb136.blacknight.com
pemlinweb137.blacknight.com
pemlinweb138.blacknight.com
pemlinweb139.blacknight.com
pemlinweb140.blacknight.com

pemmysql44-5.blacknight.com
pemmysql47-5.blacknight.com
pemmysql48-5.blacknight.com
pemmysql50-5.blacknight.com

Update 7:45 - All servers are now back online

inbound DDoS attack on pemlinweb118

2013-05-13T10:56:25Z

There is an inbound DDoS attack on pemlinweb118 , and we have been forced to take it offline until the attack ceases.

Services affected on this node are as follows
78.153.216.147
78.153.216.13

Customers will have experienced a short period of latency or slow network performance across the network during this attack.

The network has returned to stability for the moment and we will update on the status of pemlinweb118 later today

Update 12:30:
The attack has stopped and service to pemlinweb118 has resumed as normal. We are closely monitoring the situation and will give further updates should it reoccur.

emergency reboot of pemmssql01

2013-05-13T07:29:18Z

We are rebooting pemmssql01 on Wednesday 15th May at 08:00 in order to resolve an issue with backup. The reboot will take approximately 5 minutes and shared windows servers using the databases on this server will be briefly affected during the outage.

services are fully restored

pemvzlin12 drive replacement

2013-05-09T17:05:10Z

The server pemvzlin12 requires a drive replacement. It will be taken offline tomorrow morning (Friday 10th May) just after 8am for approx 30 minutes.

The following IPs will be affected:

78.153.208.104
78.153.208.126
78.153.208.131
78.153.208.135
78.153.208.140
78.153.208.212
78.153.208.22
78.153.208.231
78.153.208.247
78.153.208.45
78.153.208.60
78.153.208.70
78.153.209.102
78.153.209.150
78.153.209.188
78.153.209.190
78.153.209.219
78.153.209.233
78.153.209.58
78.153.210.161
78.153.210.197
78.153.210.224
78.153.210.70
78.153.210.8
78.153.211.102
78.153.211.131
78.153.211.135
78.153.211.16
78.153.211.165
78.153.211.19
78.153.211.216
78.153.211.32
78.153.211.44
78.153.211.46
78.153.211.47
78.153.211.59
78.153.211.60
78.153.211.61
78.153.211.64
78.153.211.70
78.153.211.95
78.153.211.99

Update: The maintenance have been completed and all services are back online

Pemvzmps42 scheduled maintenance - 7:45 AM Wednesday 8th of May

2013-05-07T17:18:11Z

Due to the frequency of the issues concerning this hardware node we will be bring it offline to facilitate a hardware replacement.

Services affected:

pemmysql07.blacknight.com

81.17.254.18

pemlinweb38.blacknight.com

81.17.254.250

The server will be unavailable for up to 60 minutes.

Update: 8:30 The maintenance have finished and server is back online

pemvzmps42 not responding

2013-05-06T02:44:02Z

The server pemvzmps42 has become non-responsive. An engineer has been dispatched to investigate.

The following servers/IPs are affected:

pemlinweb38.blacknight.com
mysql519.cp.blacknight.com

81.17.254.18
81.17.254.250

Update 05:30

The server required an FSCK which has now completed and the services are back online.

Legacy Shared Hosting Server Ragnell issue

2013-05-03T20:58:44Z

Legacy shared hosting serer ragnell (81.17.252.110) is currently having issues which is causing it to become unresponsive. We are investigating the issue and will update once we have more information.

Update 23:00 Server is now back online

Shared hardware node pemvzmps28 has become unresponsive

2013-05-02T17:53:40Z

Shared hardware node pemvzmps28 has become unresponsive and we are currently investigating the issues
sites hosted on pemlinweb25 / 256 are affected

81.17.254.64
81.17.255.104
81.17.255.20
81.17.255.21
81.17.255.26
81.17.255.51

we will update this post when we have more information

UPDATE 20:11

A compromised website was running perl processes which was causing the load to run high. We have stabilised the server now

Pemvzmps42 scheduled maintenance - 8:00 AM Friday 3rd of May

2013-05-02T09:29:36Z

The server will be rebooted on Friday Morning for a routine maintenance. Services affected:

pemmysql07.blacknight.com

81.17.254.18

pemlinweb38.blacknight.com

81.17.254.250

The server will be unavailable for approximately 30 minutes.

Scheduled reboot of SHARED HOSTING DB SERVER PEMMSSQL01

2013-05-01T13:31:27Z

We will be rebooting pemmsql01 this evening Wednesday May 1 17:00 to resolve an issue with the backup software . The server should be back within 5 minutes.
We will update the post if there are any issues

Network Maintenance

2013-04-30T17:38:48Z

As part of our network upgrade program, we will be relocating one of our core backnet switches to a new rack on Saturday 4th May between 10:00 and 15:00. All traffic will be moved over to another switch beforehand.

Customers may experience increased latency and packet loss for short intervals as the network re-converges.

Qmail maintenance Monday 6th of may 22:00 for up to 4 hours

2013-04-30T14:37:31Z

Summary: In order to beef up our mail handling capabilities we're upgrading our LDAP configuration. During this period customers can expect to see authentication errors for sending and receiving email.

When: Starting at 22:00 on Monday 6th of May for 4 hours

What: In order to increase our mail handling capacity we have to add extra compute power to our LDAP nodes. In addition to this we're upgrade from CentOS 4 to CentOS 5 which allows us to upgrade LDAP to a much more recent version. Also we'll be adding further replica's which will mean the system becoming more fault tolerant. It'll also mean we can add further mail servers to deal with increased load times.

Services affected:

Webmail on:

webmail.blacknight.com
altmail.blacknight.com

Pop/SMTP/Imap on:

mail.blacknight.com
imap5r.cp.blacknight.com
pop33r.cp.blacknight.com
smtp1r.cp.blacknight.com

Update: 7 May @ 00:40

All services were fully restored at 00:30.