December 2011 Archives

The Dinadan server is undergoing an emergency reboot - service should resume shortly

Update 01:57 - normal services have been resumed

pemlinweb95, pemlinweb96, pemlinweb97 and pemlinweb98 are currently offline we are investigating the issue and will update.

Servers Affected:

pemlinweb95 78.153.215.233

pemlinweb96 78.153.215.234

pemlinweb97 78.153.215.235

pemlinweb98 78.153.215.236

Update: The servers are coming back online now, we will update again once they are fully back online

Update 16:10: The servers are back online now

We are currently experiencing issues with a server from our older shared hosting platform, the linux shared hosting named Ector.

Our engineers are working on resolving this asap and will update this blog post once completed.

We're currently experiencing a very high load on our shared linux webserver, PEMLINWEB68 and PEMLINWEB67. Our engineering team are currently working to resolve this.

update   9:17 all services have returned to normal

We need to reboot VPS node pemvzwin11 to resolve some issues we are seeing on the node.

Once it comes back up we will update this status post.

The affected VPS's are:
pemvzwin11:
78.153.196.51
78.153.196.53
78.153.196.54
78.153.196.55
78.153.196.56
78.153.196.59
78.153.196.61
78.153.196.64
78.153.196.67
78.153.196.72
78.153.196.73
78.153.196.76
78.153.196.79
78.153.196.83
78.153.196.146
78.153.196.181
78.153.196.182
78.153.196.183
78.153.196.184
78.153.196.189
78.153.196.190
78.153.196.196

Update 11:00: Server is back online and All VPS's have started

We're currently experiencing a problems on our shared linux webservers, PEMLINWEB69 and PEMLINWEB70. Our engineering team are currently working to resolve this.

Update 17:15 - All services have returned to normal.

We're currently experiencing a very high load on our shared linux webserver, PEMLINWEB63 and PEMLINWEB64. Our engineering team are currently working to resolve this.

Update 15:25 - These servers are now returning to normal.

Update 15:50 - It appears that the servers are still experiencing some issues, we're currently working on the issue.

Update 16:35 - After replacing a RAID card, the services are now returning to normal again.

We're currently experiencing a very high load on our shared linux webserver, PEMLINWEB68 and PEMLINWEB67. Our engineering team are currently working to resolve this.

Update 12:45 - All services have returned to normal

As part of upgrading our core network, we're going to be upgrading part of the core network in Interxion. This will consist of the following tasks:

•     Upgrading one of the switch blades in each of our core routers in Intexion

•     Moving the connection to INEX LAN2 onto a new router.

There should be no downtime caused by this upgrade, however it is possible that there will be slight blips in connectivity as we take the core routers on and off the network.

This is a reminder status post, the original post was issued on the 14/12/11 http://www.blacknightstatus.com/2011/12/network-upgrade-201211-2300---211211-0100.html

The above Shared Hosting MySQL server is experiencing difficulties. We hope to have these resolved within the next few minutes.

Update 00:55 - the server has been rebooted, and is doing a file system check on boot.

Update 01:10 - the server has returned to normal, and is now accepting MySQL connections.

We are seeing issues with our linux shared hosting nodes PEMLINWE89/90 - our engineers are working to resolve this issue currently.

We're currently experiencing some packet latency on our shared dedicated and colocation network 78.153.200.0/23 - Our engineers are working to resolve this ASAP.

We are currently experiencing issues with Pemlinweb 67 + 68 . The node is being rebooted at the moment . Ip's affected are
78.153.215.161
78.153.214.177
78.153.214.198
78.153.214.207
78.153.215.36
78.153.215.160
78.153.214.191
78.153.214.202
78.153.214.219
78.153.214.238

We will update this post as soon as we know more

68 is back online , 67 is running an fsck and should be back shortly

update 13:44 - both servers are now online - thank you for your patience

As part of upgrading our core network, we're going to be upgrading part of the core network in Interxion. This will consist of the following tasks:

• Upgrading one of the switch blades in each of our core routers in Intexion
• Moving the connection to INEX LAN2 onto a new router.

There should be no downtime caused by this upgrade, however it is possible that there will be slight blips in connectivity as we take the core routers on and off the network.

We're currently experiencing a very high load on our shared linux webserver, PEMLINWEB68 and PEMLINWEB67. Our engineering team are currently working to resolve this.

We'll be performing upgrades on the Cloud Store between Midday and 3PM. During this time the Cloud Store will be unavailable and the control panel may not be fully available.

Due to a memory failure on one of our shared hosting linux hardware nodes, the following servers are currently offline:

The above Windows VPS hardware node is experiencing difficulties, we have dispatched a technician to take a look. We hope to have it back online as soon as possible.

Update 09:15 - the server is back online and the affected VPSs are now booting.

We're currently experiencing a very high load on our shared linux webserver, PEMLINWEB68. Our engineering team are currently working to resolve this.

Summary: This afternoon at approx 12:04 one of the hypervisor nodes in the cloud crashed. While the system was moving the handful of VMs to other nodes after this event it became unstable.

The software in question is OnApp and it appears to be quite unstable.

Current situation:

There are a lot of pending tasks in the queue that are supposed to be starting Virtual Machines. Most if not all of these appear to be defunct and will not cancel. We have a number of VMs back online and we're trying to get the rest of them on.

We're requesting a full root cause analysis from OnApp on this issue. It's a standard part of what the software is supposed to manage so we can't see it became so unstable.

Update: 14:45: All customer VMs should now be back online. OnApp want us to upgrade to the latest code base which will avoid such instability issues in the future. We are going to postpone this until next week when we'll have ample staff resources on the development side to fix any issues that might crop up.

Due to the increased demand created on PEMLINWEB03, we are going to migrate it to new hardware. This change will happen tonight, 9th of December, 2011 at 19:00 hours.

We will be rebooting the Windows VPS Node pemvzwin13 for updates at 21:00 on 5/12/11

The update will require two reboots, the downtime should be less than 30 minutes

Affected VPS's:

pemvzwin13:
78.153.196.130
78.153.196.131
78.153.196.133
78.153.196.135
78.153.196.138
78.153.196.144
78.153.196.145
78.153.196.152
78.153.196.155
78.153.196.153
78.153.196.161
78.153.196.167
78.153.196.170
78.153.196.171
78.153.196.172
78.153.196.173
78.153.196.177
78.153.196.178
78.153.196.180
78.153.196.62
78.153.197.33

Update 21:35
The server is back online and the VPS's are starting

We're experiencing intermittent issues with POP3 access to our Hosted Exchange system. Engineers are currently investigation.

Access via Outlook Web Access is unaffected.

The DirectAdmin server igraine.blacknight.ie has become non-responsive.  Engineers are investigating and the server will be rebooted shortly.

We hope for it to return in the next 10-15 minutes, although if it requires a disk check this may take longer.  We will update if this is the case.

Update @ 17:05: This server is now back up and working as expected.

We are currently expierencing a Network issue which is affecting access to our Hosted Exchange platform.

Once we have an update we will post it here.

Update 16:45 Services are now restored.

Sorry for the interruption, we will investigate the issue to find the root cause

We had to reboot igraine as it became unresponsive and we're just waiting for the boot up to finish.

Update 10:00 Server is now back online.