Blacknight Status Blog

Debian libssl/openssl vulnerability

Tue, 13 May 2008 17:10:19 +0000

The following security bulletin was issued today by Debian Linux.

http://lists.debian.org/debian-security-announce/2008/msg00152.html

This affects Debian systems and derivatives such as Ubuntu from Edgy onwards. Please install/update openssl packages on your servers where appropriate and re-generate all SSL key/cert pairs along with an openssh keys that you use for authentication purposes.

Blacknight will be apply fixes to managed customers servers and re-generating ssh keys, ssl cert cert/key pairs during the next 24 hours.

Note: This is Debian specific, CentOS, RHEL etc are not affected.

Shared Server Arthur is currently down

Sun, 11 May 2008 14:01:00 +0000

We're investigating an issue with the server "arthur" which is currently not reachable. An engineer is on their way to look at it. We hope to have it back shortly.

There is already a spare server built for "arthur" and if the drives have gone bang wallop we'll restore the latest backups to it on the new hardware.

We'll update this blog post when more information is available.

Update: 15:55

The server is back, it required manual intervention in the form of an fsck on one of the partitions which took some time to complete. We fear the disk(s) are having issues and we're going to schedule a maintenance window in the coming few days to complete the migration. We're already syncing data to the new hardware in case it falls over again.

Temperature issues in DEG, KCP.

Thu, 08 May 2008 16:18:27 +0000

Hi All,

We've noticed temperature increases in DEG. We've been informed that KCP (Kilcarbery Park) has a burst water main and this is affecting cooling.

Currently this is not service affecting. We'll update the blog once we get more information about this from DEG.

Update: 17:15

We've had a notification from DEG that they have taken measures to mitigate this increase. Over the last 15 minutes up to 17:36 We've seen a 10C drop in temperature.

Network Maintenance: Carrier Capacity upgrade

Wed, 30 Apr 2008 17:29:07 +0000

We're upgrading our Level(3) link tomorrow morning at 8:30am local time from 100Meg to 1000Meg. This will bounce our Level(3) and Packet Exchange eXpress peerings for a brief period and everything should return to normal within 60 seconds or so.

This should only cause momentary lag where some routes that currently route over Level(3) and PE eXpress will move to another carrier and then back again once the peerings come backup.

We don't expect any network downtime during this windows.

Emergency Reboot - Morgana Shared Linux Server

Tue, 29 Apr 2008 11:01:58 +0000

At 07:30 this morning, the morgana server stopped responding to all requests, and had to be rebooted. This was completed by 07:50, and the morgana server is now up and running again.

We will update this post if we find out any further information as to why it stopped responding to requests.

PEM upgrade

Mon, 28 Apr 2008 11:13:01 +0000

Hi there,

PEM(PSA as its now called) will be upgraded to version 2.7 tomorrow morning starting at 6am. PEM has some UI and Backend bugs that needed to be fixed that Parallels have included in the 2.7 release. As we're affected by 2 of these bugs we're having them update our install to 2.7 from 2.6.5.

The upgrade of our current system should take about 3 hours. Parallels development/engineering team will be doing this upgrade. There will be approx 3 minutes of downtime per Windows VPS node and upto 35 minutes of downtime to our CP.

We'll post tomorrow with an "all clear" once we've tested everything after the update.

As usual, if you notice any issues please contact us via e-mail (support at blacknight dot ie) or directly on our helpdesk.

VPS servers memory management change

Wed, 23 Apr 2008 11:30:30 +0000

Hi there,

In order to change how we manage VEs memory in Virtuozzo we need to restart the Virtuozzo service on each hardware node (for linux VPS only).

We've made all the necessary config changes already. We'll be doing the restarts just before 6pm Irish Time. (2hours from this post time.)

Switch Replacment

Mon, 21 Apr 2008 08:56:21 +0000

The switch in DEG which caused issues has to be replaced tonight as it is starting to misbehave again. This will be done at around 20:00 tonight, and should only cause a minor outage as servers are moved from the old switch to the new switch.

Switch Issue in DEG causing loss of network connectivity for some customers

Tue, 15 Apr 2008 09:04:17 +0000

Hi there,

1 shared server is down and several customer machines are currently off air due to a switch fault in DEG.

The switch's interfaces are up, but it has stopped passing frames. We'll post updates as we get them.

Update: 13:15

Switch has been rebooted and access has returned for all customers. We'll examine our syslogs etc and see what we can find out. We've replacement hardware onsite already so if this re-occurs we'll replace this switch in the coming 24 hours.

Billing System Upgrade Email

Mon, 07 Apr 2008 06:38:17 +0000

We informed you some time ago of an upcoming upgrade to our billing / control panel system.

You will probably receive an email from us later today regarding our billing system.

The text of the email is below:

Dear XXX

As you may know we are currently deploying a new billing and control panel system.

In order to facilitate the migration we would appreciate your help.

Please follow the link below to create your new control panel login.

NB: We will not ask you for any personal information at any time.

To create your new control panel login, please visit

Should you encounter difficulties please contact support@blacknight.com

--
Regards
Team Blacknight
http://www.blacknight.com

Once you get this email you will be able to setup your new login for our new billing system which will be going live very soon.
If you have any queries or issues please let us know.
NB: This change affects ALL Blacknight clients

NTL/UPC routing issue

Fri, 04 Apr 2008 05:26:44 +0000

We're getting notifications about NTL routing issues. This covers NTL, Chorus Ireland customers only. UK appear unaffected.

We're unaware of any network problems on our end (green lights all over)

We've opened tickets with some of our carriers to see if they can shed any light on it.

ORDB Update

Sun, 30 Mar 2008 08:45:51 +0000

We have been contacted by several clients who are having issues sending and receiving emails to various ISPs and companies.

We at Blacknight ARE NOT using ORDB as it is defunct, however it is obvious that quite a few people still are. We cannot assist you, as the mail servers affected are not under our control.

If you have issues sending mails please check the error message. If there is a reference to ORDB you will need to contact the person / company you are sending the mail to.

DNSBL ordb.org causing issues for people

Wed, 26 Mar 2008 11:50:56 +0000

Some of you might remember relays.ordb.org going offline a couple of years ago. It was one of the widely used, free, DNS Blacklists.

Yesterday at some point they started them backup and they appear to have blacklisted every single IP address.

Our monitoring system checks all configured IPs on all nodes against blacklists of this kind and sends you a notification via email or sms if one of your IPs gets on a list.

Please ignore these, but check all your mail servers to make sure they are not blocking based on "relays.ordb.org".

.com whois server issues

Wed, 26 Mar 2008 08:55:23 +0000

Over the past couple of days there have been intermittent issues with the whois server we use to check domain availability for com/net/org/info/biz

This should now be resolved

New Home For Blacknight Maintenance Notifications

Mon, 17 Mar 2008 22:55:58 +0000

This will be the new home for Blacknight service status updates and notifications.

We're currently tweaking the settings and customising the layout etc., so please excuse the current look and feel.

We'll be posting to the main blog once this site is ready.

In the meantime feel free to subscribe to the RSS feed