We are planning an upgrade of a switch that is utilized by some of our dedicated and co-location customers. The plan includes upgrading the 24 port switch to a new 48 port switch.
The upgrade will take place on Wednesday the 11th of November at 22:00 hours.
All dedicated and co-location clients with equipment in the cabinet
shall be notified separately via email to ensure they are fully up to
speed on the matter.
The new switch will be mounted and connected and then the customers simply migrated over one by one to the new switch. There should be a blip of no more than 5/10 seconds per port.
This upgrade will allow for expansion in the future.
Once the upgrade is complete I will post a status update.
Thank you for your patience and understanding.
Update 10:16PM - The switch migration was completed. The downtime per port was aprox 10 seconds.
Summary: At 09:04 we had a 30 second blip in InterXion. This happened during the process of bringing a series of new switches online. Ordinarily this operation is non service affecting however for some reason it caused a problem on a number of vlans.
All service was restored on or before 09:05 and the case is closed.
We are currently experiencing a network DDoS attack. We are working to block the attack asap and will update this post with regular status updates.
Update 11.05: The situation is currently back to normal operations, the issue has been blackholed and the routers are showing normal loads again. The engineering team is monitoring the network closely.
Update 11.41 The issue has been tracked back to one of our peers causing floods of traffic on our transit routers. Engineers are currently blacklisting the errorous traffic.
Update 11.49 The network has stabilized again. We are monitoring it closely.
We will be doing some maintenance on our core network in Interxion. We have to move some connections in order to facilitate the introduction of extra firewalls at a future date.
As all the connections are redundant, there should no interruption to the network, however there could be two slight blips are the network figures out the new paths.
Update 26/09/09 07:05: This work has been completed with no interruption to the network.
Summary: Shared hosting server morgana is currently experiencing issues and our engineering team is working on it.
We'll post updates when we have them.
Update: 17:00
Sorry about the delays in getting back to you all. This machine is dead, we're currently in the process of building a replacement machine. We hope to have it ready in the coming hours, at which point it'll take some time to migrate the data off of the disks. We estimate approx 8 or so hours for the complete restore. We do apologise for the delays in this, however these machines have upward of 10m+ files on them and these can take some time to restore.
UPDATE: 19:12
The new hardware is online and the technical team are currently working on setting software and services and transferring data onto it.
Update: 23:10
The new server is up. All mysql data is restored, all configuration is restored. The main data store is currently at 65% restored totalling around 52GB completed. Restores can take some time as there's a lot of data involved.
We've tested serveral websites and they all appear to be working fine.
On the upgrade, it's a more modern server than the old morgana was and has 8GB of ram. So expect to see some additional performance gains.
Update 01:00
This machine is now backup. All sites that we've checked appear to be working normally, e-mail is flowing etc. If you have any issues please let our helpdesk know.
We are currently experiencing severe issues on our network due to some form of denial of service attack
Our technical team are working on it and we will update you as soon as we know more
Update 13:30: We've had the DDOS blocked by 2 carriers so far and we're hoping the rest will get it sorted shortly.
Update: 13:40: All carriers have blocked the DDOS for us at this stage. It was targeted at 1 IP address of one of our VPS customers. Having discussed this with the customer it looks like their VPS was hacked and this may have attracted the DDOS.
Status as of 13:50: All Clear
We experienced an issue today where one of our transit routers had an issue due to the carriers router flapping it's bgp session. We've shut down this BGP session and had a conversation with the carrier about the issue. While bgp was doing it's thing, some people may have noticed some latency or slowness to connect. This was due to the routes moving from 1 ISP to the other.
Currently Global Cross and Cogent are carrying our traffic. Level(3) are out of the picture for the moment.
Further updates will be posted as we have them.
Update: 17:05
Packet Exchange have acknowledged an issue on their network. One of their customers was advertising the Level(3) router IP into the same VLAN we're in. So Level(3) and this other customer were fighting for the IP. This session is still down.
As an interim measure to ensure we continue to offer the same high level of service that our customers are used to we're putting Tiscali back into the loop. So we'll be back to 3 live carriers. This is happening now and Tiscali should be starting to take traffic away from Cogent and Global Crossing.
Further updates will be posted until this issue is resolved.
Update: 14:55 May 21st
Packet Exchange report that this issue is resolved. We've also received an RFO. With this in mind we'll be turning Level(3) back up at 18:30 this evening. This will cause some brief latency while routes re-converge. A final update will be posted once our engineering team confirm everything is ok once we bring this circuit back up.
Update: 20:30 May 21st
Level(3) has been live for the past 2 hours and all is looking well. This is the last update on this particular ticket. We're closing it now.
At around 14:40 today we began to notice a slow down in connections to parts of our firewalled network in the InterXion data centre. This can affect any of our new shared hosting plans, VPS plans, and some dedicated and colocated equipment.
Our engineers are working on what may be causing the network congestion and the symptoms range from either a slow connection to your website, VPS, or server, or a temporary loss in connection entirely.
This is not affecting all services in this data centre and it is not a total loss of connectivity, there is just congestion that is slowing down the traffic into some parts of the firewalled network that protect the above systems.
Summary: At approx 20:12 this evening we had a network event that caused us to loose peering with Packet Exchange eXpress and Cogent. This represents approx 60% of our external network traffic to the internet. We were able to see it originated from 1 VLAN in Data Electronics and we're working to find a cause.
Connectivity is restored and has been for some time, but we've had reports of patchy connectivity.
Update: A sequence of events caused a number of our internal and external peers to drop during a 8 minute window last night.
The events went as follows:
20:12 Global Crossing peering goes down, re convergence begins and our route reflectors recalculate best routes to the internet.
20:12 30 seconds after the GC event, our Cogent peering on the same router flaps, again we have our route reflectors re calculating the best routes to the internet.
20:14 One of our Core routers in InterXion bounces it's internal iBGP peerings and some OSPF peerings. This caused another cascaded recalculation of routes and at this point one of our route reflectors in DEG crashed. At this time all traffic traversing DEG (as this was the live router) stopped routing, approx 30 seconds later it's BGP partner took the load and traffic started to flow again.
20:15 several INEX peerings on LAN#2 flapped and we also saw 13 flaps of our peerings with the Packet Exchange eXpress route servers.
All the above BGP peering flaps caused reconvergence within our network. This made it look from some aspects that the network was down, but the internet and ourselves were just figuring out the best path in and out to us. The network stablised @ 20:20 and has been fine since.
We're investigating further for the route cause of this issue.
When: 22:00 on Wed March 18th until 23:59 on Wed March 18th.
Summary: Our cp.blacknight.com infrastructure currently runs on 10/100 switches with GigE uplinks to our core network. Two such racks are going to have new Cisco 3560G-48 switches put in to replace the existing switches. The newer racks already have GigE switches in them so those will be unaffected.
Outage Notification:
The following services will be affected during this upgrade:
Qmail hosting / Webmail
Windows shared hosting
Linux shared hosting
cp.blacknight.com
Windows VPS Servers (pemvzwin04/05 are unaffected)
Linux VPS servers (pemvzlin06/07 are unaffected)
ns1/ns2.blacknight.com (staged change over, ns1 first then ns2)
None of the services should be affected for more than 30 seconds at a time, as the new network infrastructure will be in place before the move is done so it's just a matter of re-patching the equipment.
As usual if you have any queries please contact our support team.
Update: March 18 20:20
Just a reminder about this maintenance window. All our planning and work behind the scenes means it'll go ahead as scheduled.
Update: March 18 11:20
This work is pretty much completed.We had a few hiccups with 1 or 2 new cables not pushing home correctly. Everything should be back now, thanks for your patience.