Network connectivity issues

Message

Summary: At approx 20:12 this evening we had a network event that caused us to loose peering with Packet Exchange eXpress and Cogent. This represents approx 60% of our external network traffic to the internet. We were able to see it originated from 1 VLAN in Data Electronics and we're working to find a cause.

Connectivity is restored and has been for some time, but we've had reports of patchy connectivity.

Update: A sequence of events caused a number of our internal and external peers to drop during a 8 minute window last night.

The events went as follows:

20:12 Global Crossing peering goes down, re convergence begins and our route reflectors recalculate best routes to the internet.
20:12 30 seconds after the GC event, our Cogent peering on the same router flaps, again we have our route reflectors re calculating the best routes to the internet.
20:14 One of our Core routers in InterXion bounces it's internal iBGP peerings and some OSPF peerings. This caused another cascaded recalculation of routes and at this point one of our route reflectors in DEG crashed. At this time all traffic traversing DEG (as this was the live router) stopped routing, approx 30 seconds later it's BGP partner took the load and traffic started to flow again.
20:15 several INEX peerings on LAN#2 flapped and we also saw 13 flaps of our peerings with the Packet Exchange eXpress route servers.

All the above BGP peering flaps caused reconvergence within our network. This made it look from some aspects that the network was down, but the internet and ourselves were just figuring out the best path in and out to us. The network stablised @ 20:20 and has been fine since.

We're investigating further for the route cause of this issue.