We are currently experiencing a network DDoS on our shared hosting networks. Our engineers are working to mitigate the attack ASAP.
More information will be posted as soon as it becomes available.
03:32 - This issue has been fully resolved.
May 2011 Archives
Summary: Currently the LDAP services that provide auth and delivery information to our qmail cluster is experiencing issues. We're working to resolve that now.
Additionally we'll have more servers for the cluster in a few days which should take the heat off the cluster and provide more stability for everyone.
Update 13:30: Due to the sheer number of retries by people there's an effective DOS (deniel of service) occurring on the LDAP servers. We're working on this now to rate limit connections.
Update 13:37: We've got the queries under control now. Services should be coming back online for everyone. If you have any issues past 13:45 please let us know.
Additionally we'll have more servers for the cluster in a few days which should take the heat off the cluster and provide more stability for everyone.
Update 13:30: Due to the sheer number of retries by people there's an effective DOS (deniel of service) occurring on the LDAP servers. We're working on this now to rate limit connections.
Update 13:37: We've got the queries under control now. Services should be coming back online for everyone. If you have any issues past 13:45 please let us know.
Update 14:48: After extensive monitoring all looks to be running smoothly. We're sorry for the outage.
Summary: We're currently under attack from a DOS targeted towards one of our customers. We're working with our carrier partners to mitigate the attack.
We'll post further info when it's available.
UPDATE 1038
Our technical team have been working with our upstream providers to both mitigate the issues and find the target of the attack
At present all transit is up and working. Our monitoring systems are showing an "all clear" across the network
In order to avoid these kind of issues we had already planned a network upgrade some time ago. This will be happening in the next 10 to 14 days as we install more network equipment.
We'll post further info when it's available.
UPDATE 1038
Our technical team have been working with our upstream providers to both mitigate the issues and find the target of the attack
At present all transit is up and working. Our monitoring systems are showing an "all clear" across the network
In order to avoid these kind of issues we had already planned a network upgrade some time ago. This will be happening in the next 10 to 14 days as we install more network equipment.
We are currently experiencing network issues.
Our technical team are working on this situation as quickly as possible
Our technical team are working on this situation as quickly as possible
Summary: At 15:34 the shared hosting server Igraine has had a kernel panic and has had to be rebooted. We suspect this to be a backup related issue. As such we're going to disable the during business hours snapshot for this server.
It is currently back up and running and one of our engineers is monitoring it.
It is currently back up and running and one of our engineers is monitoring it.
Summary: the mysql server known to you as mysql359 or mysql359int is currently down. We're working on a fix for this node now and we hope to have it back up and running before 09:00.
What's affected:
Mysql connections from any web server to:
mysql359.cp.blacknight.com
mysql359int.cp.blacknight.com
Update: 09:08 this node is back online since 08:59. The issue looks to be a deadlocked file system during the nightly backup. The result of this is that lots of processes were hung in "D" state which means waiting for disk. We don't see it too often so it's not a major concern. However we will probably take this node down for maintenance in the not too distant future to do further checks on the disks.
What's affected:
Mysql connections from any web server to:
mysql359.cp.blacknight.com
mysql359int.cp.blacknight.com
Update: 09:08 this node is back online since 08:59. The issue looks to be a deadlocked file system during the nightly backup. The result of this is that lots of processes were hung in "D" state which means waiting for disk. We don't see it too often so it's not a major concern. However we will probably take this node down for maintenance in the not too distant future to do further checks on the disks.
PEMLINWEB 65 and 66 are currently unresponsive. An engineer is looking into the issue and this post will be updated once we have more information.
UPDATE 23:25: The root cause of the downtime turned out to be a failed drive in the raid, which unfortunately took out the whole box. The bad drive has been replaced, and the RAID is currently resyncing. In order to give the RAID a chance, pemlinweb65 and 66 are being kept off for another 20 mins.
UPDATE 00:00: Further investigation shows another drive not too happy. To be on the safe side, this drive will also be replaced one the RAID is rebuilt. The RAID rebuild currently has another 15-20 minutes left to go.
UPDATE 00:30: The RAID is currently rebuilding, but being slowed down a bit by PEMLINWEB 65 and 66 running a quota check when starting up. Hopefully everything should be back up and running in the next 30 mins.
UPDATE 01:15: Both PEMLINWEB65 and 66 are back up and running.
UPDATE 23:25: The root cause of the downtime turned out to be a failed drive in the raid, which unfortunately took out the whole box. The bad drive has been replaced, and the RAID is currently resyncing. In order to give the RAID a chance, pemlinweb65 and 66 are being kept off for another 20 mins.
UPDATE 00:00: Further investigation shows another drive not too happy. To be on the safe side, this drive will also be replaced one the RAID is rebuilt. The RAID rebuild currently has another 15-20 minutes left to go.
UPDATE 00:30: The RAID is currently rebuilding, but being slowed down a bit by PEMLINWEB 65 and 66 running a quota check when starting up. Hopefully everything should be back up and running in the next 30 mins.
UPDATE 01:15: Both PEMLINWEB65 and 66 are back up and running.
Summary: This node has stopped all it's containers and when they're being started virtuozzo throws an error. We're reporting this issue to Parallels for investigation. However in the mean time we have restarted the node and all the containers on this node are now started again.
Note: There are limited ways to see if a virtuozzo node is functioning properly. i.e. that containers are working properly. We're going to investigate ways of monitoring what's going on within virtuozzo to make sure we can spot these things earlier.
Note: There are limited ways to see if a virtuozzo node is functioning properly. i.e. that containers are working properly. We're going to investigate ways of monitoring what's going on within virtuozzo to make sure we can spot these things earlier.
We are currently experiencing issues with one of our older linux shared hosting boxes, igraine.blacknight.ie - Our engineers are working to resolve this issue asap.
We will update this post once completed.
16:41 - This issue is now resolved
We are currently experiencing an issue with the File Manager that is accessed through cp.blacknight.com on our shared Linux hosting packages. We have raised this with the control panel vendors and hope to have the issue resolved as soon as possible.
In the meantime there should be no issue with accessing your files via FTP if needed and you can find your FTP details in cp.blacknight.com under:
Web Space > FTP Access
Update 12:25: This issue should be fully resolved now.
In the meantime there should be no issue with accessing your files via FTP if needed and you can find your FTP details in cp.blacknight.com under:
Web Space > FTP Access
Update 12:25: This issue should be fully resolved now.
Summary: At 16:55 this evening a customer added a sub domain to our control panel which appears to have caused some sort of problem for our deployment system. In turn our deployment system has set all files/folders in our vhosts (1) directory to be owned by this users webspace. Fixing this issue is not trivial but we're working on it with our software vendor in order to get a fix asap.
What's affected: Any website, sub domain or web application hosted on pemlinweb19 or the following IP addresses:
81.17.254.79 81.17.254.227 81.17.254.236 81.17.255.7
1) the vhosts directory is where each webspace gets created as a numeric number that matches the Webspace ID in your control panel. e.g. /vhosts/100001/ the 100001 here would show in your cp.
Update 20:15: We've restored the default file permissions to everyone's webspaces. People whom are using CMSs with Apache mode php may have to request that we change permissions for them. Alternatively if you change your webspace to use php5 in cgi mode you shouldn't have any problems.The root cause of the issue is known and it's a bug in our control panel. Thankfully the customer who triggered the issue was in touch with us immediately and won't do what they did again.
What's affected: Any website, sub domain or web application hosted on pemlinweb19 or the following IP addresses:
81.17.254.79 81.17.254.227 81.17.254.236 81.17.255.7
1) the vhosts directory is where each webspace gets created as a numeric number that matches the Webspace ID in your control panel. e.g. /vhosts/100001/ the 100001 here would show in your cp.
Update 20:15: We've restored the default file permissions to everyone's webspaces. People whom are using CMSs with Apache mode php may have to request that we change permissions for them. Alternatively if you change your webspace to use php5 in cgi mode you shouldn't have any problems.The root cause of the issue is known and it's a bug in our control panel. Thankfully the customer who triggered the issue was in touch with us immediately and won't do what they did again.