Results tagged “Network upgrade” from Blacknight Hosting Network Status

When: 22:00 on Wed March 18th until 23:59 on Wed March 18th.

Summary: Our cp.blacknight.com infrastructure currently runs on 10/100 switches with GigE uplinks to our core network. Two such racks are going to have new Cisco 3560G-48 switches put in to replace the existing switches. The newer racks already have GigE switches in them so those will be unaffected.

Outage Notification:

The following services will be affected during this upgrade:

Qmail hosting / Webmail
Windows shared hosting
Linux shared hosting
cp.blacknight.com
Windows VPS Servers (pemvzwin04/05 are unaffected)
Linux VPS servers (pemvzlin06/07 are unaffected)
ns1/ns2.blacknight.com (staged change over, ns1 first then ns2)

None of the services should be affected for more than 30 seconds at a time, as the new network infrastructure will be in place before the move is done so it's just a matter of re-patching the equipment.

As usual if you have any queries please contact our support team.

Update: March 18 20:20

Just a reminder about this maintenance window. All our planning and work behind the scenes means it'll go ahead as scheduled.

Update: March 18 11:20

This work is pretty much completed.We had a few hiccups with 1 or 2 new cables not pushing home correctly. Everything should be back now, thanks for your patience.

We have been experiencing  some network issues this afternoon resulting in packet loss

Our technical team are working on a full resolution and a more detailed report will be published as soon as it is available

Update: 16:40

The issue is resolved and we're investigating the problem. We believe a compromised machine was sending our large volumes of UDP traffic that was causing the packet loss. Once we've got further information we'll send it out.

Update: 16:55

We've narrowed this issue down to the HA firewall pair in DEG that protects certain IP ranges running out of sessions per second. Currently this equipment is a HA pair of Cisco 5520 ASA firewall appliances. They have a maximum session per second count of 280k.

bk1-fw3# sh conn count
185559 in use, 280000 most used

The MAX used above is the MAX this firewall pair can do. We're ordering another pair that is similarly configured and we'll move some networks off this pair. This will prevent this issue from occuring again.

However *note* the networks hanging off of this firewall are considered at risk until we've put in new hardware. If you have any questions please let us know.

We're upgrading our Level(3) link tomorrow morning at 8:30am local time from 100Meg to 1000Meg. This will bounce our Level(3) and Packet Exchange eXpress peerings for a brief period and everything should return to normal within 60 seconds or so.

This should only cause momentary lag where some routes that currently route over Level(3) and PE eXpress will move to another carrier and then back again once the peerings come backup.

We don't expect any network downtime during this windows.