January 2009 Archives

We're working on some upgrades on our helpdesk this evening, so it will not be fully available for the next couple of hours.

This will also affect the FAQ / Knowledgebase, as they run of the same system

UPDATE: Sorry about not updating sooner. The system was back online fully after about an hour.

Due to a number of factors outside our control we are not offering LiveChat for technical support today.

We are offering it for sales and billing queries.

The above named hardware node is currently not functioning correctly. We've already initiated a reboot of it and the VPS should be back up in 15-20 minutes.

To rectify this issue, we'll be moving the VPS off of this node over the coming 24 hours and we'll then investigate these issues further without affecting Customers.

We'll post an update shortly.

Update: 16:15

This Node is now back online and 80% of the VPS, are started. The remaining 20% should be started in the next 10 or so minutes.

Our support and sales desk (online ticketing system) will not be available this evening from 7pm until 9pm.

We are upgrading the server hardware that we use which means that some downtime is unavoidable.

Any tickets / emails submitted during the downtime should be processed after the system comes back online

UPDATE: Due to software issues we have had to postpone this upgrade

We have been having transient issues with the Blacknight storefront for the last hour.

Currently we are looking into these with a hope to finding a permanent fix.

If you encounter issues while placing an order please let us know

Note: This has zero impact on any hosted services

Update 1507 : This issue has been escalated internally with our software vendor
Update 1607: We have established the cause of the issue and have taken action to mitigate it. Things should be back to normal in the next couple of minutes

We've installed the latest versions of Joomla and Wordpress in our application vault. When you view your application in the CCP, it'll say there's an upgrade available. Click on the application and then click on the Upgrade button. The system will do the upgrade behind the scenes.

Any questions please address them to the support team.

One of the things upgraded this morning in POA HF02 was courier-imap on the qmail cluster machines. Seemingly there's an issue currently with IMAP, pop3 remains working at least. We're investigating the IMAP issues and we'll hopefully have a resolution sooner rather than later.

This applies to all IMAP connections made to mail.blacknight.com

Update: 11:12am

Webmail + IMAP should be back now. We don't know why this issue cropped up this morning, but it looks to be related to the nfs servers clock being off by 4 minutes and IMAP doesn't like the time skew. We've setup ntpdate to run every day on the NFS servers now so this problem shouldn't reoccur.

Tomorrow morning starting at 4am our deployment team will roll out upgrades to the Windows and Linux sitebuilders. This will upgrade all installs to Sitebuilder version 4.5 and it'll also upgrade all templates.

We don't expect this to cause any issues, but like any upgrade it might cause some knock on effects.

Please e-mail support if you have any difficulties.

Summary: Windows node pemvzwin02 is experiencing issues with internal virtuzzo API calls. This is causing a task backlog in our provisioning system. In order to diagnose the issue further we've been asked to reboot the node.

We're taking the opertunity to run windows updates while we do this, so at least some good will come of it.

The following list of VPSs are affected:

vps-238.cp.blacknight.com, vps-258.cp.blacknight.com, vps-262.cp.blacknight.com, vps-265.cp.blacknight.com, vps-271.blacknight.com, vps-282.cp.blacknight.com, vps-287.cp.blacknight.com, vps-288.cp.blacknight.com, vps-302.cp.blacknight.com, vps-317.cp.blacknight.com, vps-331.cp.blacknight.com, vps-337.cp.blacknight.com, vps-341.cp.blacknight.com, vps-342.cp.blacknight.com, vps-343.cp.blacknight.com, vps-344.cp.blacknight.com, vps-346.cp.blacknight.com, vps-347.blacknight.com, vps-353.cp.blacknight.com, vps-354.cp.blacknight.com, vps-355.cp.blacknight.com, vps-357.cp.blacknight.com, vps-361.blacknight.com, vps-362.cp.blacknight.com, vps-386.cp.blacknight.com, vps-387.cp.blacknight.com, vps-390.cp.blacknight.com, vps-407.cp.blacknight.com, vps-409.cp.blacknight.com, vps-410.cp.blacknight.com,
vps-584.cp.blacknight.com, vps-589.cp.blacknight.com, vps-612.cp.blacknight.com, vps-620.cp.blacknight.com, vps-626.cp.blacknight.com, vps-635.cp.blacknight.com, vps-642.cp.blacknight.com, vps-645.cp.blacknight.com, vps-664.cp.blacknight.com, vps-667.cp.blacknight.com, vps-672.cp.blacknight.com, vps-684.cp.blacknight.com, vps-689.cp.blacknight.com, vps-704.cp.blacknight.com, vps-709.cp.blacknight.com,
vps-710.cp.blacknight.com, vps-716.cp.blacknight.com, vps-718.cp.blacknight.com, test.zerotouch.ie

I'll update this ticket once it comes back up and all VPSs are back up. It'll probably take upto an hour, but it's completely necessary.

Update: 21:30

The node rebooted successfully and about 70% of the VPSs have restarted. The rest should be back online in the next 10 to 15 minutes or so.

We're continuing our investigation of the issues we've been encountering in our backend systems.

Tomorrow morning our software vendor will be patching our back end provisioning systems. This will affect multiple customer facing systems including cp.blacknight.com, access to plesk and virtuozzo power panel on Plesk loaded VPSs and potentially other systems. All sites, services will remain in place, just admin access to customers will not be available during the upgrade.

It should be starting around 4am and will last for upto 1 hour. I'll post an update tomorrow morning post upgrade with news of fixes to issues we've been having.

Update: 22:05 - 5th Jan 09

Due to issues beyond our control this has been re-scheduled until Thursday 8th at the same time.

Update: 09:50 - 8th Jan 09

This upgrade has been installed and so far we've seen no issues. A number of back end system bugs have been resolved and also a few shared hosting issues.