January 2011 Archives

We are currently experiencing issues with our linux shared web server: pemlinweb31 (81.17.254.38). Our engineers are working to resolve the issue and will keep this post updated with relevant info.

Friday 1740

The store / shop is currently offline for maintenance.

At present we are not processing any orders and DNS updates are also not available.

Update 21:15: The store is back online as of 20:12, all other services associated with Domain registrations such as contact updates, dns updates etc are all back to normal aswell.

We have been alerted to a fault occurring within the power distribution unit which powers the following direct admin servers:

Cador - 81.17.252.196

Ragnell - 81.17.252.110

Balin - 81.17.252.15

As we treat this with extreme urgency, we are planning the removal of the current PDU and replace it as soon as we can to ensure maximum level of service to you.

We are planning an outage window as follows:

DATE: Wednesday, 2nd of Feb.

TIME: 23:00 - 00:00 hours. (This is the maintenance window the downtime will most likely not last this duration).

Blacknight aspire to providing the best possible service to our customers however equipment can fault from time to time.

Summary: There's a spike in inbound e-mail which our mail servers are taking a bit of time to proceeds. The volumes we're talking about is around 4-5 times the normal mail volume.

This is causing some delays in e-mail delivery both locally and remotely. The delay in e-mail should shorten over the next 60 to 90 minutes. Obviously we're closely monitoring the situation and dealing with spam as we can.

We are currently experiencing issues with the hardware node which hosts PEMLINWEB03 and PEMLINWEB10.

Summary: The node pemvzlin03 is experiencing erratic high load at the moment. It's perfectly responsive however the virtual machines appear to be experiencing some issues.

We're going to reboot the node in the next few minutes to ensure that the VMs are working correctly. We're aware that this is the third wednesday in a row where this has happened and we have a fair idea of what is causing it so we're gathering as much information as possible and going back to one of our software vendors to see can they give us some insight into the issue.

Update 08:44: As we could actually login to this node unlike the last occurrence of this issue (as a result of last weeks software update) we were able to power cycle the node using ipmi. We're going to speak to Dell today about firmware updates for the bios and the raid card. The main reason for doing so is that it's during high IO situations i.e. backups running where this problem occurs.

All VPS servers are now starting up and should be back by 09:00

We are currently experiencing issues with our shared hosting linux web server, Ector. Our engineers are working on resolving this currently and will update this post with any updates.

We are currently experiencing issues with two linux shared hosting web servers pemlinweb59 and pemlinweb60.

Several users have reported issues logging in to the control panel this morning and our technical team are working on the issue to resolve it as quickly as possible.

UPDATE 1010 - it should be working now, though if anyone has any issues please let us know.

We are currently experiencing issues with PEMLINWEB03 and PEMLINWEB10.

The hardware node which hosts pemlinweb47(78.153.214.38) and pemlinweb48(78.153.214.39) is currently unresponsive. An engineer is on site and getting it back up at the moment.

Update 13:27: Both servers are on the way up now and should be ready in the next few minutes.

Telnic, the registry operator for the .tel domain name extension, are conducting maintenance on the TelHosting platform.

When? Wednesday 26nd January 2011 at 07:00 UTC

The Telhosting system and its API will be offline for about 15 - 30 minutes (possibly less)

Summary: One of our legacy shared hosting machines Priamus is experiencing issues at the moment.

We've deployed an engineer to investigate and we expect it back by 13:00

Update 13:00: The server is currently doing an disk check and it should be back in the next 10-15 minutes. Approx 13:15.

Summary: In order to diagnose the issues causing instability in this node we're apply all available patches and kernel updates. In order to get the latest kernel running we obviously have to reboot the node. As this reboot will be done properly i.e. it won't be a power cycle all the virtual machines on it should start up pretty quickly.

We estimate around 30 minutes downtime for all VMs during this window.

When: 22:30 this evening Wednesday 19th of Jan 2011 we'll be doing this reboot.

Update 22:40: This work is complete. There was around 10 minutes or so of downtime.

Summary: This machine has become unresponsive so we'll be rebooting it. As this is the second week in a row that it's had issues we'll be putting a notification up later today where we'll take it down and apply some updates to the virtualisation software in order to resolve this issue.

All the VPS on this node are unresponsive currently.

Update 09:37: the node is now booting. It'll take approx 30 minutes for of the VPS to come back up fully.

Update 10:38: All the containers on this node are now fully started. it took a little longer than expected.

Summary: Due to a large amount of inbound e-mail (from newsletters and other sources) there's a backlog of mail delivery on the mail servers at the moment. It's only a few 100 e-mails above normal but it'll take a bit of time for this back log to clear.

As this mail is being delivered to local mailboxes NFS use is up around 30% on an average day and this is causing some delays for IMAP and POP3 users also when checking their e-mail.

We're almost ready to deploy new hardware that should hopefully put this to bed for a time until a more permanent solution can be put in place.

The .je and .gg domain name registry has informed us that they are currently conducting emergency maintenance.

No new registrations or updates will be possible while this work is being carried out.

Existing .je and .gg domain names will continue to resolve as before.

Our Windows VPS platform will be unavailable for a period of approximately 2 hours from 22:00 GMT on Tuesday next, 18th January.

This outage is necessary to both update the Windows VPS platform, and to improve its backup strategy.

All Windows VPSs will be unavailable for a period of about 30 minutes during this outage.

 

Update 23:30 :- This outage window has completed, and all VPSs are currently starting, if they are not back online already.

There was a hardware problem with a server a small number of VPSs are on, but we are currently migrating these to a different hardware node. The IP Addresses of the effected VPSs are as follows:

78.153.208.59
78.153.208.251
78.153.209.196
78.153.208.160
78.153.208.217
78.153.209.232
78.153.209.148
78.153.209.161

PEMVZLIN03 is currently unresponsive along with all VPS on it. An engineer is already on site checking the server. We will have it back up as soon as possible.

Affected VPSs:

Summary: In order to install some vital software on this hardware node we need to reboot it. It will take around 15 minutes or so to complete the reboot.

When: Monday 10/1/2010 @ 22:30 hours

The following containers / VEIDs are on this node and will be affected.

 VEID  |   ip address   
-------+-----------------
  1126 | 078.153.208.057
  1162 | 078.153.208.086
  1166 | 078.153.208.091
  1169 | 078.153.208.094
  1177 | 078.153.208.098
  1181 | 078.153.208.111
  1189 | 078.153.208.067
  1550 | 078.153.209.136
  2216 | 078.153.208.155
  2361 | 078.153.211.069

Update 22:50: All VPS are back online after a successful reboot.

The .it registry operator will be conducting maintenance on some parts of their network later today, from 1300 to 1400 CET.

During this period there will be interruptions to a variety of services including domain lookups.

Existing registered .it domain names will not be impacted.

ns2.blacknightsolutions.com in Germany is currently down. We're in contact with the local provider in order to get it back up as soon as possible. ns.blacknightsolutions.com is responding to queries as normal.

The .info and .org registry operators will be conducting maintenance on 15 January 2011 from 1500 to 1900 UTC

During this period the following services will be impacted:

• WHOIS
• New registrations
• Updates

Existing .info and .org domain names will resolve as normal

We experienced a DDoS on our network this morning which caused some network latency and packet drops. It commenced just at 5AM GMT and was resolved by 08.20AM. The DDoS was directed at one anycast customer.

The IEDR, the registry operator for the .ie ccTLD, has been having technical issues this morning.

The situation is being worked on by their technical team, but many services impacting domain registration and updates may be impacted.

These services include:

- WHOIS

- API (which we use to send updates to the registry)

- IEDR website

This Windows VPS node has lost all networking connectivity during a diagnostics test and has been rebooted remotely. All VPSs on this node will be down for a period of approximately 10 minutes.

 

We apologise for any inconvenience this may cause.

Summary: Some customers behind one set of HA firewall pairs are currently experiencing slowdowns intermittently. This is due to an on-going DDOS attack.

We're working on this at the moment.

Services affected:

Shared Linux Hosting + Webmail + SMTP + POP3/IMAP (78.153.214.0/23, 81.17.254.0/23), Shared Windows Hosting (81.17.250.0/23) and Hosted Exchange.

The apparent effect is slowness or slow down when viewing websites.

Due to the high volume of mailboxes and traffic to our ever expanding mail cluster, we need to do some improvements. This time it will be on one of the servers that physically holds the mailboxes.

The registry operator for the .ie ccTLD (IEDR) has informed us of upcoming scheduled maintenance on Tuesday January 11th 2011 from 1800 to 2000

During this period the following services will be offline:

• IEDR website
• Whois
• IEDR control panel
• Registrar API

This means that no new registrations or updates will be possible during this timeframe

Existing .ie domain names should not be impacted.

We have just experienced an issue with our frontend server that handles our billing. This server is the public facing and would display all of your billing information etc via the control panel.