The server "ragnell" is currently experiencing issues.
From what we have been able to determine one site on the server has caused issues.
Our engineering team are currently working on restoring services as quickly as possible to affected clients.
UPDATE: The server has been restored to full service.
The "DEG Mesh", i.e. the network that Data Electronics provide to customers has been experiencing issues intermittently during the day today. We've escalated this issue to them and we're awaiting a response regarding the issue.
Currently it's not affecting DNS, but DNS responses may take a few 100ms longer than normal.
Update: 9:45
The issue appears to be resoled. We've had no formal notification of a fix as of yet. I'll give an update once we've received a formal RFO.
We weren't aware of the maintenance window that they were doing last night and it looks like they have overrun by several hours.
Currently we're running on Tiscali primarily but we've Cogent and Global Crossing to fall back onto if we have to. You'll probably see traffic going over Global Crossing as we added a GigE from them yesterday.
There are no issues currently that we can see as a result of Level(3) being down for us. We've more than adequate capacity elsewhere in the network to cover us.
Information:
Currently Blacknight have 4 x GigE connections to the outside world delivered in 2 diverse locations in Dublin.We've multiple wave lengths between both of these locations with secondary backup links where needed. Our network core is spread between these 2 data centres comprising of 12 Juniper routers using iBGP, eBGP, route reflector technology and OSPF in our core to provide the ultimate experience in network traffic flow. Additionally all our inter-datacentre layer 3 links run over resiliant layer 2 paths where we use RPVST for redundancy. We also use VRRP for all our customer facing traffic where there is no firewalling and where there is firewalling we use HA pairs of Cisco ASAs.
Lastly in adition to the above we've 1 x GigE connection to INEX and 1 x 100Meg connection to INEX which gives our Irish Users the experience of us being on their ISPs network. We are one of the few HSPs in Ireland who are directly peered with all Irish ISPs.
Due to issues we've been experiencing with our Paypal integration we have decided to disable it temporarily.
We hope to resolve the issue today, but it may take several days.
Needless to say we still accept all other payment methods
UPDATE: This issue has been resolved
This morning our backup CWDM link from DEG <-> InterXion experienced an outage of 5 seconds approx. We got a call immediately from our fibre provider informing us of a possible low light situation with this link.
We've a ticket open regarding this and we're monitoring the situation at the moment. Due to the design and resilience built into our network this outage did not affect production traffic.
Update: 12:30
We got a call back regarding this. The fibre strand carrying our wave lengths was snagged while pulling other fibre into the HBA facility in North Dublin. This resulted in a 25db loss in signal strength. This snag was found and the fibre was straightend and the problem should be fully resolved. No reoccuring blip occurred during the fix.
On Thursday morning starting at 7am we'll be upgrading our CP system to PBA 4.3 from 4.2. This upgrade will take between 1 and 2 hours and for the duration of it the Customer Control Panel (https://cp.blacknight.com) will be unavailable.
You'll be able to access your VPS via SSH or Plesk/Vzpp by going to it's IP address and port 8443 or 4643 using https.
If you have any questions please let us know.
The following security bulletin was issued today by Debian Linux.
http://lists.debian.org/debian-security-announce/2008/msg00152.html
This affects Debian systems and derivatives such as Ubuntu from Edgy onwards. Please install/update openssl packages on your servers where appropriate and re-generate all SSL key/cert pairs along with an openssh keys that you use for authentication purposes.
Blacknight will be apply fixes to managed customers servers and re-generating ssh keys, ssl cert cert/key pairs during the next 24 hours.
Note: This is Debian specific, CentOS, RHEL etc are not affected.
We're investigating an issue with the server "arthur" which is currently not reachable. An engineer is on their way to look at it. We hope to have it back shortly.
There is already a spare server built for "arthur" and if the drives have gone bang wallop we'll restore the latest backups to it on the new hardware.
We'll update this blog post when more information is available.
Update: 15:55
The server is back, it required manual intervention in the form of an fsck on one of the partitions which took some time to complete. We fear the disk(s) are having issues and we're going to schedule a maintenance window in the coming few days to complete the migration. We're already syncing data to the new hardware in case it falls over again.
Hi All,
We've noticed temperature increases in DEG. We've been informed that KCP (Kilcarbery Park) has a burst water main and this is affecting cooling.
Currently this is not service affecting. We'll update the blog once we get more information about this from DEG.
Update: 17:15
We've had a notification from DEG that they have taken measures to mitigate this increase. Over the last 15 minutes up to 17:36 We've seen a 10C drop in temperature.