Network Connectivity

This issue has been fully resolved and all network connectivity has been restored.

Here is full timeline of the events that occurred yesterday:

18:30 - An attack begun against one of our shared linux web serves, pemlinweb04.blacknight.com. This was not picked up with our monitors are first due to the low volume of attack traffic

19.45 - The attack began to increase, sending huge amounts of UDP traffic to pemlinweb04.blacknight.com. At this point alerts began to filter through and the on-call engineer was alerted.

20.00 - The issue became apparent that because of the sheer volume of UDP traffic the network switch located in the rack where pemlinweb04 was located was exhausting itself and dropping traffic to all devices connected in the rack being facilitated by that switch.

20.15 - The engineers disabled the site the traffic was being directed to and began null routing the sources the traffic was coming from.

20.25 - A noticeable drop in the traffic began showing causing the switch to resume normal operations. 

20:40 - All services were returned to normal operation.

Windows VPS Nodes

Shared Windows Hosting Nodes:

pemwinweb07

pemwinweb08

pemwinweb09

pemwinweb10

Shared Linux Hosting Nodes

pemlinweb01

pemlinweb04

pemlinweb05

pemlinweb07

pemlinweb19

pemlinweb20

pemlinweb21

pemlinweb22

pemlinweb23

pemlinweb24

MySQL DB Nodes

mysql71.cp.blacknight.com