Results tagged “Level(3)” from Blacknight Hosting Network Status

We experienced an issue today where one of our transit routers had an issue due to the carriers router flapping it's bgp session. We've shut down this BGP session and had a conversation with the carrier about the issue. While bgp was doing it's thing, some people may have noticed some latency or slowness to connect. This was due to the routes moving from 1 ISP to the other.

Currently Global Cross and Cogent are carrying our traffic. Level(3) are out of the picture for the moment.

Further updates will be posted as we have them.

Update: 17:05

Packet Exchange have acknowledged an issue on their network. One of their customers was advertising the Level(3) router IP into the same VLAN we're in. So Level(3) and this other customer were fighting for the IP. This session is still down.

As an interim measure to ensure we continue to offer the same high level of service that our customers are used to we're putting Tiscali back into the loop. So we'll be back to 3 live carriers. This is happening now and Tiscali should be starting to take traffic away from Cogent and Global Crossing.

Further updates will be posted until this issue is resolved.

Update: 14:55 May 21st

Packet Exchange report that this issue is resolved. We've also received an RFO. With this in mind we'll be turning Level(3) back up at 18:30 this evening. This will cause some brief latency while routes re-converge. A final update will be posted once our engineering team confirm everything is ok once we bring this circuit back up.

Update: 20:30 May 21st

Level(3) has been live for the past 2 hours and all is looking well. This is the last update on this particular ticket. We're closing it now.

At approx. 20:35 Level(3) full route set dissappeared on one of our GigEs. Customers may have experienced slight delays as BGP decided which carrier to use for all those routes.

We've opened a ticket with Packet Exchange and we're awaiting an answer from them. We're currently using Tiscali and Global crossing (out of DEG and IX) with Cogent as backup if either fail.

We'll post an update once we hear more from them on this issue.

Update: Saturday Aug 30th 12:10am

We've had no update as of yet from Packet Exchange however the issue appeared to get fixed already. When we get the RFO from them, we'll post a summary.

We weren't aware of the maintenance window that they were doing last night and it looks like they have overrun by several hours.

Currently we're running on Tiscali primarily but we've Cogent and Global Crossing to fall back onto if we have to. You'll probably see traffic going over Global Crossing as we added a GigE from them yesterday.

There are no issues currently that we can see as a result of Level(3) being down for us. We've more than adequate capacity elsewhere in the network to cover us.

Information:

Currently Blacknight have 4 x GigE connections to the outside world delivered in 2 diverse locations in Dublin.We've multiple wave lengths between both of these locations with secondary backup links where needed. Our network core is spread between these 2 data centres comprising of 12 Juniper routers using iBGP, eBGP, route reflector technology and OSPF in our core to provide the ultimate experience in network traffic flow. Additionally all our inter-datacentre layer 3 links run over resiliant layer 2 paths where we use RPVST for redundancy. We also use VRRP for all our customer facing traffic where there is no firewalling and where there is firewalling we use HA pairs of Cisco ASAs.

Lastly in adition to the above we've 1 x GigE connection to INEX and 1 x 100Meg connection to INEX which gives our Irish Users the experience of us being on their ISPs network. We are one of the few HSPs in Ireland who are directly peered with all Irish ISPs.

We're upgrading our Level(3) link tomorrow morning at 8:30am local time from 100Meg to 1000Meg. This will bounce our Level(3) and Packet Exchange eXpress peerings for a brief period and everything should return to normal within 60 seconds or so.

This should only cause momentary lag where some routes that currently route over Level(3) and PE eXpress will move to another carrier and then back again once the peerings come backup.

We don't expect any network downtime during this windows.