On Wednesday April 1st at 4am, we'll be upgrading our billing system to version 4.4 Service Pack 1. This will introduce a significant number of bug fixes and feature enhancements.
This will require some downtime, so the following systems and customer facing interfaces will not be available during this window:
1) our online store (https://store.blacknight.com).
2) the billing tab in cp.blacknight.com when you're logged in
3) domain name server changes, contact updates etc
Note: This affects all customers who have their billing handled by https://cp.blacknight.com
The maintenance will start at 4am approx and complete shortly before 8am.
Update: April 1st 09:45
We're having some unexpected problems in a few areas:
Firstly our payment plugin for the billing system isn't currently working, so can't accept debit/credit card payments. These tested fine on our test environment.
Secondly the plugin that handles all interaction with the IEDR (for .ie domains) is currently not working. Again this tested fine in our test environment, but not in the live. We're working with Parallels to get both plugins backup and running.
Lastly and not as critical as the problems above, there are some CSS/layout issues on the customer and admin facing parts of the billing system. The interface looks a little weird and it's not too easy to navigate. Hopefully this will be sorted soon.
More updates as we have them.
We have been informed that the data centre which hosts one of our secondary nameservers (ns2.blacknightsolutions.com) is conducting maintenance this Friday night.
The server will be physically powered down and moved to a different part of the data centre in Germany.
We have not been given an exact timescale for the downtime, but we are expecting anything from two to four hours.
We will update you if / when we have any more details
NB: This will not affect any accounts hosted on the new hosting system which use ns1.blacknight.com / ns2.blacknight.com
Summary: At approx 20:12 this evening we had a network event that caused us to loose peering with Packet Exchange eXpress and Cogent. This represents approx 60% of our external network traffic to the internet. We were able to see it originated from 1 VLAN in Data Electronics and we're working to find a cause.
Connectivity is restored and has been for some time, but we've had reports of patchy connectivity.
Update: A sequence of events caused a number of our internal and external peers to drop during a 8 minute window last night.
The events went as follows:
20:12 Global Crossing peering goes down, re convergence begins and our route reflectors recalculate best routes to the internet.
20:12 30 seconds after the GC event, our Cogent peering on the same router flaps, again we have our route reflectors re calculating the best routes to the internet.
20:14 One of our Core routers in InterXion bounces it's internal iBGP peerings and some OSPF peerings. This caused another cascaded recalculation of routes and at this point one of our route reflectors in DEG crashed. At this time all traffic traversing DEG (as this was the live router) stopped routing, approx 30 seconds later it's BGP partner took the load and traffic started to flow again.
20:15 several INEX peerings on LAN#2 flapped and we also saw 13 flaps of our peerings with the Packet Exchange eXpress route servers.
All the above BGP peering flaps caused reconvergence within our network. This made it look from some aspects that the network was down, but the internet and ourselves were just figuring out the best path in and out to us. The network stablised @ 20:20 and has been fine since.
We're investigating further for the route cause of this issue.
Our linux shared hosting box known as "Bors" is currently experiencing difficulties. One of our engineers is looking at it right now and we hope to have a quick resolution.
We'll post further updates once we have more information.
Update: 17:26
This machine has kernel panic'd. That's two in a week, we suspect that perhaps the memory has become faulty. We'll follow up with a maintenance window next week to replace the memory.
When: 22:00 on Wed March 18th until 23:59 on Wed March 18th.
Summary: Our cp.blacknight.com infrastructure currently runs on 10/100 switches with GigE uplinks to our core network. Two such racks are going to have new Cisco 3560G-48 switches put in to replace the existing switches. The newer racks already have GigE switches in them so those will be unaffected.
Outage Notification:
The following services will be affected during this upgrade:
Qmail hosting / Webmail
Windows shared hosting
Linux shared hosting
cp.blacknight.com
Windows VPS Servers (pemvzwin04/05 are unaffected)
Linux VPS servers (pemvzlin06/07 are unaffected)
ns1/ns2.blacknight.com (staged change over, ns1 first then ns2)
None of the services should be affected for more than 30 seconds at a time, as the new network infrastructure will be in place before the move is done so it's just a matter of re-patching the equipment.
As usual if you have any queries please contact our support team.
Update: March 18 20:20
Just a reminder about this maintenance window. All our planning and work behind the scenes means it'll go ahead as scheduled.
Update: March 18 11:20
This work is pretty much completed.We had a few hiccups with 1 or 2 new cables not pushing home correctly. Everything should be back now, thanks for your patience.
As next Tuesday, March 17th 2009, is St Patrick's Day our offices will be closed.
We will be offering technical support between 10am and 4pm.
If you have a dedicated server or colocation please refer to the "out of hours" contact details.
An issue is occurring where people use the feature "leave a copy of this message on my email server for X number of days" in their e-mail client. If our pop3 servers for what ever reason are not available for a period of time and during this period of time your e-mail client gets an error message it's left in an inconsistent state. So to ensure that you've got all of your e-mail, it'll download the mail again.
The reason behind this issue is that our mail servers are using NFS to access the shared storage and they're leaving stale locks on files which is causing issues. We've got a temporary fix in place that can fix this in a few short seconds, however the duplicate e-mail issue can occur when this happens.
To fix this we're going to move to NFSv4, currently we're using NFSv3 which has locking issues and is generally not anywhere near as good as NFSv4. This solution has to be tested fully before we can put it live. We hope to test it in our lab over the coming days and I'll do a maintenance window to schedule a few minutes downtime for switching the live servers over.
Due to a customer compromise our firewall pair in DEG for our old shared hosting and some shared dedicated and colo was up and down for a period of 11 minutes.
Time of outage: 21:50
Time of service restore: 22:01
Cause: Compromised machine flooding the inside interface with udp packets.
Fix: shut down of port of compromised machine
We are currently investigating an issue affecting some of our email servers in the shared cluster.
If this issue affects you rest assured that we are working on it and will post updates as soon as we can
Last update 12:20
This issue is resolved now. An NFS locking issue which has been intermittently causing new e-mails to not download via POP3 has been found and resolved. We consider this case closed.
The @Mail install at webmail.blacknight.com is being upgraded to the latest version in order to fix a few UI bugs.
The webmail will be blocked for the duration of the upgrade, which should be less than 30 minutes.
Anyone using IMAP or POP to access their email accounts will be unaffected.
Update: 16:55
This upgrade has taken place and is now complete, webmail should be working again.
Summary: March 3rd @ 4am to 8am we'll be upgrading our new shared hosting platform and some back end services.
About 75 bugs are going to be fixed tomorrow morning starting at 4am. The result of this upgrade will mean that cp.blacknight.com and some other services will be down during the upgrade.
The maintenance window from parallels is approx 4 hours, so it should be fully completed by 8am.
A list of affected services and some well known bugs are below:
1) Hosted exchange bug fixes
2) linux shared hosting bug fixes
3) windows shared hosting bug fixes
4) shared e-mail, there will be some upgrades to our qmail servers which will cause pop3/imap issues during the maintenance window.
5) Bug#264931 upgradeSubscription for container if capacity VPS Number per HW node is over - currently if the limit we've set per hw node is reached and a customer gets an upgrade, their VPS will be shutdown and migrated to another node. This has been resolved.
6) Bug#271059 Webmail login does not work for 'webmail.yourdomain'
Update: March 3rd @ 09:45
This updated was delayed for a few hours but is currently being performed. It should be completed by 10:15 at the latest.
Update 2: 1253 - All services should have returned to normal before 11am. However if you are experiencing issues that you think may be related please let us know.
