May 2009 Archives

At approximately 2pm this afternoon one of the power strips in one of our cabinets in Interxion "tripped".

As most of the servers in the rack are either dual powered or using the other side's PDU the issue wasn't spotted immediately.

Once the issue was identified the data centre staff reset the trip and power was restored.

All affected clients have been contacted or attempted to be contacted.

The following servers will be rebooted between 6:00 and 7:00 am on Thursday May 28th.

pemlinweb01 through pemlinweb12

This is to facilitate some memory upgrades due to spikes in activity during peak hours. We'll be doing them in batches starting with:

pemlinweb01.blacknight.com
pemlinweb03.blacknight.com
pemlinweb05.blacknight.com
pemlinweb10.blacknight.com

then:

pemlinweb02.blacknight.com
pemlinweb04.blacknight.com
pemlinweb06.blacknight.com
pemlinweb11.blacknight.com

then:

pemlinweb07.blacknight.com
pemlinweb08.blacknight.com
pemlinweb09.blacknight.com
pemlinweb12.blacknight.com

The reboots should be quite quick as the memory install should only take 30 seconds per node.

If you have any issues after 07:00 on the 28th please contact support immediately.

NB: All times are Dublin, Ireland. This service notice only affects shared servers. It has no impact on VPS servers

Update: 07:00

Phase 1 and 2 complete with all nodes back online. An unforeseen set back has delayed the startup of pemlinweb7/8/9/12. Once an fsck is done on these machines they should be back up. It's around 25% of the way done. We estimate that they'll be back by approx 07:30. I'll post further updates shortly. This maintenance window has been extended for another hour as a precaution.

Update: 07:35

Those machines are now back online. This maintenance window is now completed and closed.

We experienced an issue today where one of our transit routers had an issue due to the carriers router flapping it's bgp session. We've shut down this BGP session and had a conversation with the carrier about the issue. While bgp was doing it's thing, some people may have noticed some latency or slowness to connect. This was due to the routes moving from 1 ISP to the other.

Currently Global Cross and Cogent are carrying our traffic. Level(3) are out of the picture for the moment.

Further updates will be posted as we have them.

Update: 17:05

Packet Exchange have acknowledged an issue on their network. One of their customers was advertising the Level(3) router IP into the same VLAN we're in. So Level(3) and this other customer were fighting for the IP. This session is still down.

As an interim measure to ensure we continue to offer the same high level of service that our customers are used to we're putting Tiscali back into the loop. So we'll be back to 3 live carriers. This is happening now and Tiscali should be starting to take traffic away from Cogent and Global Crossing.

Further updates will be posted until this issue is resolved.

Update: 14:55 May 21st

Packet Exchange report that this issue is resolved. We've also received an RFO. With this in mind we'll be turning Level(3) back up at 18:30 this evening. This will cause some brief latency while routes re-converge. A final update will be posted once our engineering team confirm everything is ok once we bring this circuit back up.

Update: 20:30 May 21st

Level(3) has been live for the past 2 hours and all is looking well. This is the last update on this particular ticket. We're closing it now.

The shared, DirectAdmin, server "Ragnell" is currently experiencing issues.

Our technical team are working on a resolution.

This notification only affects sites on that server.

UPDATE: 1940 As the server is unresponsive a reboot is required.

UPDATE: 2027 Service was fully restored a few minutes ago.

At around 14:40 today we began to notice a slow down in connections to parts of our firewalled network in the InterXion data centre.  This can affect any of our new shared hosting plans, VPS plans, and some dedicated and colocated equipment.

Our engineers are working on what may be causing the network congestion and the symptoms range from either a slow connection to your website, VPS, or server, or a temporary loss in connection entirely. 

This is not affecting all services in this data centre and it is not a total loss of connectivity, there is just congestion that is slowing down the traffic into some parts of the firewalled network that protect the above systems.

We've added a few more screencasts to our YouTube channel covering:

• Editing dns entries
• Creating / Managing MySQL databases
• Setting up cronjobs (Linux)

They are also embedded on the Wiki

We recommend viewing the HD version where possible, as the image quality is a lot better

The Windows VPS node pemvzwin04 requires an emergency reboot, which will be carried out at 23:00 this evening. All VPSs hosted on this node will be effected. Downtime will be in the region of 20 minutes.

We apologise for any inconvenience this may cause.

Affected VPSs:

78.153.209.65
78.153.209.106
78.153.209.226
78.153.209.175
78.153.210.18
78.153.210.16
78.153.210.20
78.153.210.22
78.153.210.23
78.153.210.27
78.153.210.28
78.153.210.30
78.153.210.18
78.153.210.36
78.153.210.37
78.153.210.38
78.153.210.42
78.153.210.53
78.153.210.54
78.153.209.95
78.153.210.69
78.153.209.253
78.153.210.70
78.153.210.76
78.153.210.80
78.153.210.88
78.153.210.46
78.153.210.121
78.153.210.126
78.153.210.132

Today we've installed the follow pear modules and drivers on all our linux shared hosting plans on our new system.

i.e. any linux package where you login to cp.blacknight.com to administer it, has these features available.

On php4:

MDB2 + mysql driver

On php5:

MDB2 + mysql + mysqli drivers

This is effective immediately.

Our Qmail cluster was experiencing some issues earlier this morning.

For an unknown reason which is still being  investigated, qmail stopped closing connections correctly and eventually ran out of available connections causing the outages.

Forcefully killing off the stuck qmail processes and restarting qmail fixed the issues, and it's still being monitored

UPDATE: 12:24

On further investigation our technical team found some other issues, which would have impacted end users

In essence LDAP had issues. As LDAP looks after all authentication then other services would have appeared to be degraded.

Although we may not reply to all negative comments that people make about us in public, we do listen to what people are saying about us.

One area which people have expressed concern, disappointment and frustration over the last few months has been in relation to documentation for the various services we offer.

In order to help address this we recently launched a wiki so that both our staff and clients could share tips, tricks, tutorials, howtos etc.,

At the moment there isn't a huge amount of content, but it is growing.

Areas covered include:

• VPS
• Plesk
• Domain Registration
• Shared Hosting
• Control Panels

And anything that people want it to cover...

We chose MediaWiki to run the system, so if anyone wants a particular extension installed or other option enabled please do let us know

All feedback is welcome