Recently in network Category

We're currently experiencing some packet latency on our Dedicated and VPS network. Our engineers are working on this issue.

The issue is now resolved and network connectivity is returning to normal.

As part of upgrading our core network, we're going to be upgrading part of the core network in Interxion. This will consist of the following tasks:

•     Upgrading one of the switch blades in each of our core routers in Intexion

•     Moving the connection to INEX LAN2 onto a new router.

There should be no downtime caused by this upgrade, however it is possible that there will be slight blips in connectivity as we take the core routers on and off the network.

This is a reminder status post, the original post was issued on the 14/12/11 http://www.blacknightstatus.com/2011/12/network-upgrade-201211-2300---211211-0100.html

We're currently experiencing some packet latency on our shared dedicated and colocation network 78.153.200.0/23 - Our engineers are working to resolve this ASAP.

As part of upgrading our core network, we're going to be upgrading part of the core network in Interxion. This will consist of the following tasks:

• Upgrading one of the switch blades in each of our core routers in Intexion
• Moving the connection to INEX LAN2 onto a new router.

There should be no downtime caused by this upgrade, however it is possible that there will be slight blips in connectivity as we take the core routers on and off the network.

Summary: We're currently experiencing some network latency on our shared hosting networks. Our engineers are working to resolve this issue ASAP.

We're currently experiencing high amounts of traffic flooding into our network. Our engineers are working to resolve this currently.

We're currently experiencing a network issue which appears to be affecting large parts of our shared / VPS network.

Our technical team are aware of the issue and are working on it

We will update with more information as soon as we have it

UPDATE 2300 - the issue appears to be a DDOS against our DNS servers which is affecting any service that uses them.

UPDATE 0005 - Our network engineers are working on the issue. Due to the nature of the attack it is impacting sites and services on our shared network ie. more than DNS.

UPDATE 0017 - our main site, control panel and most client sites *should* be accessible now. We're confirming any outstanding issues

UPDATE 1135 - The network issue was fully resolved last night. One or two clients reported issues this morning that were unrelated.

With respect to the issues last night I am still waiting on a more detailed explanation from our technical team, but in simple terms what happened was as follows

Something / someone launched a large attack against us sending a very large amount of DNS queries / packets of data. The huge spike in queries basically maxed out the firewalls that protect that segment of our shared hosting network.

Once our technical team were able to identify the network block that was being hit they were able to take actions to remedy the situation.

All shared hosting services were functional again from around midnight last night. We are aware of one Windows webserver that had an issue last night, but it is, as far as I know, unrelated.

If we can get a more detailed explanation of last night's issue we'll post it here / edit the post

Summary: We're currently experiencing a network problem. We're working to diagnose it and get everything that is down backup.

Initial findings: We've found some MAC related log entries in some of our core network switches. These highlight a layer 2 loop within the network. These began @ 12:48. The knock on effect was layer 2 port flaps followed by BGP and OSPF flaps localised to that data centre. Any traffic passing through that data centre via Cogent, INEX LAN2 in or out would have been dead in the water.

At 12:56 the flaps stopped and the routers began to stabilise. It took a few minutes after this for everything to calm down. We're still investigating this right now and we'll post more information as it's available.

RFO: The reason for this outage was due to an ethernet loop within our network in the InterXion datacentre. A new piece of equipment was introduced to the network earlier this week fully configured and as such it caused no issues. While this equipment was being worked on it's configuration was wiped and upon reboot it appeared on the network and caused a network loop. This caused a spanning tree event in our core switching fabric in InterXion. The result was the network outage that was observed. Typically events like this can't occur and we put strict provisions in place to prevent it however in this instance a 3rd party piece of equipment caused the issue. It wasn't immediately evident that this would occur as the device in question had been tested in our lab for 4 weeks prior to it's deployment within the data centre. Obviously we take every precaution when working on our network and sometimes events like this can occur. However it should not happen in future and we're working on ways to prevent it happening again. At the very least we hope to contain issues to individual racks rather than the entire ethernet fabric in that data centre.

We are currently experiencing a network DDoS on our shared hosting networks. Our engineers are working to mitigate the attack ASAP.