November 2010 Archives

Summary: Nov 29th @ 11:00 The mail servers are struggling to deal with the number of pop3/imap requests being asked of them this morning. We estimate that there'll be another week or so, certainly 3 days+ before the new storage platform will be ready.

Currently pop3/imap connections may time out or not connect at all for brief moments. Please ensure that you have your e-mail client set to check every 5 or 10 minutes instead of every 1 minute. This will ensure that mail works more smoothly for everyone.

We'll be announcing a time to change to the new storage during this week. Best estimates are this Friday night or Saturday night the 10th or 11th of December.

We have just lost power in our Carlow office at present which has caused us to shutdown the telephone system before the UPS runs out of power.

This will effect all telephone services with Blacknight over the regular office numbers. The on-call number is unaffected.

We expect to have power restored ASAP of course.

In the meantime if any issues arise regarding Support, Sales, Accounts etc please do not hesitate to send in a ticket to our teams who will be happy to deal with them. This can be done here.

UPDATE 09:30AM: All services are now resumed. Thank you for your patience.

PEMVZMPS48 has become unresponsive and an engineer has been sent to look at it. The following linux web servers are affected:

• pemlinweb03.blacknight.com (81.17.254.80)
  
• pemlinweb10.blacknight.com (81.17.254.92)

More updates will be posted here when available.

Update: 0900 - The server has been rebooted and all services have returned to normal.

Telnic are upgrading the Telhosting system from 0700 UTC on December 1st 2010.

The system will remain online during the upgrade, but there may be some session loss ie. you may lose your connection etc.,

Summary: There's an on going issue where by the load on the mail servers is spiking quite frequently and causing timeouts in your e-mail client.

As we've been working on a solution to this for sometime there is no immediate fix. As an interim work around if customers change their mail checking limit from 1 minute to 7-10 minutes while we get a proper working resolution for this issue this will back off the load on the servers.

We are currently experiencing some networking issues on our shared hosting network. Our engineers are working on this at the moment to resolve it asap. 

This server is having problems at the moment, so we have had to reboot it.

Normal service will resume in the next few minutes.

 

Update: We are taking the opportunity to apply some security updates, so the downtime is taking a little longer than expected. We hope to have the server back up and running as soon as possible.

PEMVZMPS48 has become unresponsive and an engineer has been sent to look at it. The following web servers are affected:

• pemlinweb29.blacknight.com (81.17.254.24)
  
• pemlinweb30.blacknight.com (81.17.254.25)

More updates will be posted here when available.

Update 15:25: The server has been rebooted and both web servers are now back up and running. We're currently investigating what bought the server down.

Summary: As part of our continued efforts to provide the best services, control panel etc to our customers we're upgrading our billing system tonight / tomorrow morning (Nov 9th / Nov 10th) starting from around 3am.

When: Starting at 3am Irish time on Wednesday 10th of November until approx 9am.

What: Our store / billing system will be offline during this process. So you won't be able to buy new domains, update contacts on domains, manage name servers etc. You will however still be able to login to cp.blacknight.com and manage your hosting, e-mail, SharePoint and VPS servers.

This is a _major_ change in our software. It's moving our backend database from MySQL to PostgresSQL and lots of User Interface changes. There'll be a new payment wizard to allow people to pay for renewal orders etc. Screen shots of this were put up on our main company blog here.

This upgraded was originally scheduled for October 27th but we're finally going ahead tomorrow after many long hours of bug fixing and testing.

UPDATE 1300
We are still working with our software vendor to fix some of the outstanding bugs.
Existing orders and plans / subscriptions are working fine. Payments are all working without issue
Unfortunately at present we are having issues processing new orders.

Update November 11th 0929
Apologies for the lack of updates. We have been working with our developers and the software vendors through the night to resolve the outstanding bugs and issues. We believe that they have all been resolved at this juncture, however if you have any issues please let us know (and provide as much detail as you can!)

Summary: In the last week to 10 days we've noticed the odd Magento site getting compromised. In each case Magento was extremely out of date and allowed the attacker to gain full access of the site and the database. Obviously if you're storing customer information in your Magento store this is a potentially huge problem.

We recommend that anyone not running the latest stable Magento install (manual installs) and the latest available install from our Appvault that you upgrade immediately.

Steps to do this:

1) take a backup of your files + database
2) check your backup to see if there were any odd looking files left behind
    e.g. htaces.php,logo.php etc
3) upload the relevant patch / files
4) ensure your site is working properly

We recommend that everyone who uses Magento does this as soon as possible.

We are currently experiencing abnormally high load on the mailservers that handle email through our cp.blacknight.com hosting platform.  That is if your email is through any of the following servers:

mail.blacknight.com
smtp1r.cp.blacknight.com
imap5r.cp.blacknight.com
pop33r.cp.blacknight.com


Email is getting delivered to the servers without issue, however the high load is preventing users from being able to log in to access your email accounts.

Our engineers are working on this at the moment and we hope to have it resolved as soon as possible.

Update: 10:50 This appears to be ldap related so we've re-indexed the ldap database and we believe this is resolved. It will take upto 30 minutes for it to settle as there are a lot of logins / second at the moment.

The .com and .net registries are undergoing scheduled maintenance on Sunday November 14th 2010 from 0100 to 0145

During this period we will be unable to process updates or new registrations.

Existing .com domain names will continue to resolve as normal

The Windows VPS hardware node pemvzwin04 will have to be rebooted at 18:00. All VPSs hosted on this Hardware Node will have to be rebooted, we apologise for any inconvenience this may cause.

An issue has arisen with the linux shared hosting server PEMVZMPS06. An engineer is being dispatched to the data center to resolve it.

We are currently experiencing issues with two of our linux shared hosting nodes:

One of our core philosophies has always been to run a "clean" network where feasible.

With that in mind we have recently started contacting clients whose websites appear to be compromised with malicious content.

We are using Google's Safe Browsing Alerts service to track malicious activity across our entire network (as members of RIPE we run our own network and are not relying on a 3rd party)

One of the best ways to avoid these kind of issues is to keep any software on your site (and your computers) up to date and change your passwords regularly.

Related articles

• Google Launches Phishing Alert Service (securityweek.com)
• Google Adds Phishing URL Alerts Into Network Tool (nytimes.com)
• Google To Warn Admins Of Malware Infestations (informationweek.com)
• Google rolls out phishing URL alerts for admins (go.theregister.com)