In order to resolve an issue we are seeing on the hardware node PEMVZWIN02 we are going to reboot it at 10:30PM on the 27th of April 2010.
The following VPSs will be affected:
78.153.208.116 VPS-238
78.153.208.123 VPS-258
78.153.208.16
VPS-262
78.153.208.127 VPS-282
78.153.208.28 VPS-287
78.153.209.211
VPS-288
78.153.208.154 VPS-317
78.153.208.168 VPS-331
78.153.208.149
VPS-337
78.153.208.170 VPS-341
78.153.208.172 VPS-342
78.153.208.169
VPS
78.153.208.171 VPS-344
78.153.209.151 VPS-346
78.153.208.15
VPS-347
78.153.208.62 VPS-353
78.153.208.176 VPS-354
78.153.208.177
VPS-355
78.153.208.179 VPS-357
78.153.208.184 VPS-362
78.153.210.75
VPS-387
78.153.208.205 VPS-390
78.153.208.218 VPS-407
78.153.208.222
VPS-409
78.153.208.223 VPS-410
78.153.209.118 VPS-620
78.153.209.160
VPS-664
78.153.209.164 VPS-667
78.153.209.176 PARTYCENTRAL
78.153.209.107
VPS-710
Estimated downtime is 10 minutes per VE.
We will update this post once completed.
UPDATE 22:48 - This has been completed.
Summary: We've noticed a steady increase in Joomla sites being hacked or being abused because they're out of date.
Notice: Please ensure that you keep your web applications up to date. This if for your good and the general good of the internet as a whole. If you don't your site may be compromised and badly hacked with the potential for loss of data. We advise that all customers update all of their applications when/if they're new updates. But this notification is specific to Joomla.
Please note that in your Joomla root directory you should only have a handful of php files by default.
CHANGELOG.php
COPYRIGHT.php
CREDITS.php
INSTALL.php
LICENSE.php
LICENSES.php
index.php
index2.php
configuration.php
Anything else is probably not supposed to be there, so please keep this in mind when you're updating your applications.
If you are ringing us today please excuse any background noise.
We currently have about 10 workmen in and around our offices working on various things which, unfortunately, involves a certain amount of banging and drilling.
Due to increased traffic we are seeing on one of our hardware nodes we are
going to migrate it to newer hardware.
The downtime
will occur tonight at 21:00, April 21st 2010
The downtime for the
affected node will be no more than 45 minutes.
The affected nodes are:
81.17.254.24 pemlinweb29.blacknight.com
81.17.254.25 pemlinweb30.blacknight.com
We
will update this post once completed.
UPDATE 21:15: This has been postponed until tomorrow the 21st of April at the same time, 21:00. An issue has arisen with the software to migrate the CTs. We need to discuss this with our software vendors.
UPDATE 21-05-2010: PEMLINWEB29 has been successfully migrated. We are going to reschedule the migration of PEMLINWEB30.
We are currently working on resolving some issues with cp.blacknight.com
Currently users are experiencing issues logging in to the control panel system
We are aware of the problem and are working on fast resolution
We are in the process of upgrading the version of virtuzzo on this hardware node. Normally these updates can happen on the fly. However due to the kernel updates that are needed on this occasion the node will need to be rebooted.
What's getting rebooted ?
The following nodes will be rebooted and will be offline for a period of 10 minutes maximum:
81.17.254.74 pemlinweb15.blacknight.com
81.17.254.75 pemlinweb16.blacknight.com
When ?
The downtime will occur at 21:00 on the 14th of April 2010.
We will update this blog post once completed.
UPDATE 21:16: This has been completed.
The hardware node hosting both these servers is having issues at the moment. An engineer in on the way in order to get things back up and running.
Update: 03:30: Both servers are now back up.
Update: 08:10 this issue seems to have re-occured. We're working on it now.
Update: 08:30 This node has been rebooted. We've identified the problem which is causing this issue. A disk has failed in the array and the backup software we use appears to have issues with the failed disk as it does block level backups. We've disabled the backup software until the disk can be replaced. A maintenance window will be put up on this site later today with a date/time for the disk replacement.
Currently the two machines (pemlinweb29/30 81.17.254.24/25) are doing disk checks and should be backup shortly.
Update 08:34: Both nodes are now back online and running smoothly. We are monitoring closely.
Ector is currently experiencing issues which have caused us to block all outside access till we get it stable again. As soon as it's fully back up and running, we'll open up outside access again.
Update: Currently waiting for the RAID to rebuild as it seems that the combination of normal access and the raid rebuilding was enough to bring down the server when we restarted it. This should be completed within the next 10 minutes.
Summary: At 21:00 on Thursday 15th of April, Data Electronics are upgrading their mesh network this week. This will affect some legacy services.
What: As part of this upgrade one of our legacy services which still runs off their network may experience a few seconds of downtime while the upstream router HA pair are swapped out. This is a physical cable re-patch which should only take at most 30 seconds. Domains running off of ns.blacknightsolutions.com and ns2.blacknightsolutions.com should not experiencing any down time but there's a slight chance of some slow DNS resolution for about 1-2 minutes while this change occurs.
Update: This has been completed successfully with no downtime at all.
We are currently aware of some users encountering issues with MySQL.
Our technical team are investigating the issue and we will try to provide more details.
The kind of error people are reporting is that their site is not able to connect to the database.
UPDATE 1:47PM: The issue was related to a network issue spotted on the internal interface of the mysql server. The issue has been resolved and we are monitoring it closely at present.
Due to issues with the software controlling the VPS' on PEMVZLIN06, we had to do an emergency reboot in order to get things back into a stable state. The hardware node has been successfully rebooted, and we're currently waiting for the last of the VPS to start up.
Due to an issue we are seeing with the service container on this hardware node, Plesk wasn't operating correctly. We have resolved this issue but a server reboot is required to ensure the fresh config is fully loaded.
The downtime will occur at 22:00 tonight the 8th of April 2010.
The downtime will be aprox 30 minutes.
The affected VPSs are:
78.153.208.116 VPS-238
78.153.208.123 VPS-258
78.153.208.16 VPS-262
78.153.208.127 VPS-282
78.153.208.28 VPS-287
78.153.209.211 VPS-288
78.153.208.154 VPS-317
78.153.208.168 VPS-331
78.153.208.149 VPS-337
78.153.208.170 VPS-341
78.153.208.172 VPS-342
78.153.208.169 VPS
78.153.208.171 VPS-344
78.153.209.151 VPS-346
78.153.208.15 VPS-347
78.153.208.62 VPS-353
78.153.208.176 VPS-354
78.153.208.177 VPS-355
78.153.208.179 VPS-357
78.153.208.184 VPS-362
78.153.210.75 VPS-387
78.153.208.205 VPS-390
78.153.208.218 VPS-407
78.153.208.222 VPS-409
78.153.208.223 VPS-410
78.153.209.118 VPS-620
78.153.209.160 VPS-664
78.153.209.164 VPS-667
78.153.209.176 PARTYCENTRAL
78.153.209.107 VPS-710
We will update this post once complete.
Edit: This reboot has been postponed until Friday 9th April at 22:00
Update 20:30 09/04/2010: The server has been successfully rebooted.
We are currently experiencing issues with the shared hosting linux server igraine.
Our engineers are working on this issue at present.
Update: 08:25 am - This machine was fully back in service by 02:30 - the boot loader was corrupt so an engineer had to physically go onsite and re-install grub before the machine would boot backup. Unfortunately this is all to common on older machines.
This case is now closed.
Summary: Tonight Friday April 2nd at 22:00 the virtual machine known as cp.blacknight.com is moving to a new hardware node. This is on advice from our software vendor and is to try and avoid resource shortages which appear to be happening which cause tomcat errors on the control panel which many customers have seen over the past 3 months.
When: Friday April 2nd @ 22:00
Expected downtime: approx 2 hours, but most likely around 1 hour or less. This will be the Control Panel Only. No hosting services will be affected by this.