Blacknight Status Blog

Network Maintenance Window - non intrustive

2008-06-17T11:20:37Z

Summary: On Sunday morning next @ 02:00 am one of our Fibre providers (Auroa) are carrying out maintenance on the backup ring that they provide to us.This will be a total cut of the fibre while they replace several joins to make them more flexible. During this 8 hour window our traffic will be on our primary link which does not go near the affected physical area and the route is direct to DEG.

If we do experience an outage on our primary ring due to this window, we've a tertiary path that we may use and this should automatically failover.

Any questions please address them to our technical team https://support.blacknight.ie/

update: Sunday June 22 @ 10:10am

This upgrade was successful, Aurora report that circa 04:30 they completed the work on the fibre that our CWDM ring runs over. We got the all clear at 10am to confirm the fibre was back in the man hole and that the window was completed successfully with no snags.

Morgana Issues

2008-06-13T10:47:13Z

The shared hosting server Morgana is currently not responding. It is being rebooted, and we waiting for it to come back up.

This post will be updated when we find out what caused the issue.

Update: The server is back up and we are now investigating to see what caused the issue.

"billing.blacknight.ie" Maintenance

2008-06-05T11:13:35Z

Our billing site will be down tonight at 8pm for routine maintenance. This should take less than a hour while we test to make sure that everything is back up and running correctly.

We will not be able to accept any orders during this period.

Update: The maintenance has been completed successfully, and billing should be back up now.

Ragnell Issues

2008-05-27T11:51:21Z

The server "ragnell" is currently experiencing issues.

From what we have been able to determine one site on the server has caused issues.

Our engineering team are currently working on restoring services as quickly as possible to affected clients.

UPDATE: The server has been restored to full service.

Eurid - Scheduled Downtime

2008-05-26T15:24:26Z

Eurid are conducting more maintenance and will be taking the registration systems offline again on Tuesday June 3rd from about 20h00 Belgian time until about 22h00.

The notification we received did not go into much detail, but it sounds like the registry backend will be offline for the duration of the maintenance window.

This window will affect EU domain registrations, transfers and updates only. No other services will be affected

ns.blacknightsolutions.com having high latency issues

2008-05-25T20:04:34Z

The "DEG Mesh", i.e. the network that Data Electronics provide to customers has been experiencing issues intermittently during the day today. We've escalated this issue to them and we're awaiting a response regarding the issue.

Currently it's not affecting DNS, but DNS responses may take a few 100ms longer than normal.

Update: 9:45

The issue appears to be resoled. We've had no formal notification of a fix as of yet. I'll give an update once we've received a formal RFO.

Level(3) currently down

2008-05-22T08:13:22Z

We weren't aware of the maintenance window that they were doing last night and it looks like they have overrun by several hours.

Currently we're running on Tiscali primarily but we've Cogent and Global Crossing to fall back onto if we have to. You'll probably see traffic going over Global Crossing as we added a GigE from them yesterday.

There are no issues currently that we can see as a result of Level(3) being down for us. We've more than adequate capacity elsewhere in the network to cover us.

Information:

Currently Blacknight have 4 x GigE connections to the outside world delivered in 2 diverse locations in Dublin.We've multiple wave lengths between both of these locations with secondary backup links where needed. Our network core is spread between these 2 data centres comprising of 12 Juniper routers using iBGP, eBGP, route reflector technology and OSPF in our core to provide the ultimate experience in network traffic flow. Additionally all our inter-datacentre layer 3 links run over resiliant layer 2 paths where we use RPVST for redundancy. We also use VRRP for all our customer facing traffic where there is no firewalling and where there is firewalling we use HA pairs of Cisco ASAs.

Lastly in adition to the above we've 1 x GigE connection to INEX and 1 x 100Meg connection to INEX which gives our Irish Users the experience of us being on their ISPs network. We are one of the few HSPs in Ireland who are directly peered with all Irish ISPs.

Eurid Registry Planned Downtime Saturday May 24

2008-05-21T19:17:25Z

Eurid, the organisation that manages the doteu namespace (.eu) has scheduled maintenance for next Saturday May 24th 2008.

According to their notification email registry services will be unavailable for most of Saturday ie. we will not be able to process any new registrations or modifications to existing domain names in dot eu.

This maintenance will not have any impact on other domains or services provided by us.

To maintain the integrity of the .eu database, registration services (both EPP and the web interface) will be unavailable during the migration. We (eurid) estimate that these services will be unavailable for up to six hours but the actual downtime may be much less

Paypal Payments Issues

2008-05-20T11:39:49Z

Due to issues we've been experiencing with our Paypal integration we have decided to disable it temporarily.

We hope to resolve the issue today, but it may take several days.

Needless to say we still accept all other payment methods

UPDATE: This issue has been resolved

Auroa Fibre Outage

2008-05-20T09:54:16Z

This morning our backup CWDM link from DEG <-> InterXion experienced an outage of 5 seconds approx. We got a call immediately from our fibre provider informing us of a possible low light situation with this link.

We've a ticket open regarding this and we're monitoring the situation at the moment. Due to the design and resilience built into our network this outage did not affect production traffic.

Update: 12:30

We got a call back regarding this. The fibre strand carrying our wave lengths was snagged while pulling other fibre into the HBA facility in North Dublin. This resulted in a 25db loss in signal strength. This snag was found and the fibre was straightend and the problem should be fully resolved. No reoccuring blip occurred during the fix.

VPS Control Panel Update

2008-05-20T08:18:43Z

On Thursday morning starting at 7am we'll be upgrading our CP system to PBA 4.3 from 4.2. This upgrade will take between 1 and 2 hours and for the duration of it the Customer Control Panel (https://cp.blacknight.com) will be unavailable.

You'll be able to access your VPS via SSH or Plesk/Vzpp by going to it's IP address and port 8443 or 4643 using https.

If you have any questions please let us know.

Debian libssl/openssl vulnerability

2008-05-13T16:10:19Z

The following security bulletin was issued today by Debian Linux.

http://lists.debian.org/debian-security-announce/2008/msg00152.html

This affects Debian systems and derivatives such as Ubuntu from Edgy onwards. Please install/update openssl packages on your servers where appropriate and re-generate all SSL key/cert pairs along with an openssh keys that you use for authentication purposes.

Blacknight will be apply fixes to managed customers servers and re-generating ssh keys, ssl cert cert/key pairs during the next 24 hours.

Note: This is Debian specific, CentOS, RHEL etc are not affected.

Shared Server Arthur is currently down

2008-05-11T13:01:00Z

We're investigating an issue with the server "arthur" which is currently not reachable. An engineer is on their way to look at it. We hope to have it back shortly.

There is already a spare server built for "arthur" and if the drives have gone bang wallop we'll restore the latest backups to it on the new hardware.

We'll update this blog post when more information is available.

Update: 15:55

The server is back, it required manual intervention in the form of an fsck on one of the partitions which took some time to complete. We fear the disk(s) are having issues and we're going to schedule a maintenance window in the coming few days to complete the migration. We're already syncing data to the new hardware in case it falls over again.

Temperature issues in DEG, KCP.

2008-05-08T15:18:27Z

Hi All,

We've noticed temperature increases in DEG. We've been informed that KCP (Kilcarbery Park) has a burst water main and this is affecting cooling.

Currently this is not service affecting. We'll update the blog once we get more information about this from DEG.

Update: 17:15

We've had a notification from DEG that they have taken measures to mitigate this increase. Over the last 15 minutes up to 17:36 We've seen a 10C drop in temperature.
]]>

Network Maintenance: Carrier Capacity upgrade

2008-04-30T16:29:07Z

We're upgrading our Level(3) link tomorrow morning at 8:30am local time from 100Meg to 1000Meg. This will bounce our Level(3) and Packet Exchange eXpress peerings for a brief period and everything should return to normal within 60 seconds or so.

This should only cause momentary lag where some routes that currently route over Level(3) and PE eXpress will move to another carrier and then back again once the peerings come backup.

We don't expect any network downtime during this windows.