October 2011 Archives

We're currently experiencing high amounts of traffic flooding into our network. Our engineers are working to resolve this currently.

We are currently experiencing issues with our legacy shared hosting linux server - Gorlois.

The above Windows VPS Hardware node requires a reboot to correct an issue.

This will be carried out at 22:00 this evening (Thursday 27/10)

The following customer VPSs will be effected:

78.153.196.130
78.153.196.131
78.153.196.133
78.153.196.135
78.153.196.138
78.153.196.139
78.153.196.144
78.153.196.145
78.153.196.152
78.153.196.155
78.153.196.153
78.153.196.161
78.153.196.167
78.153.196.170
78.153.196.171
78.153.196.172
78.153.196.173
78.153.196.177
78.153.196.178
78.153.196.180
78.153.196.65
78.153.196.62
78.153.197.33

 

We apologise for any inconvenience this may cause.

It would appear that a third party site that we were using to include some html on our main website was compromised and this resulted in the main Blacknight website being flagged by Google as containing malware. We have removed the included code from our website and we're currently awaiting review by the Google webmasters team and we hope to have an update and the problem resolved shortly.

Update: 08:10: this also affects the control panel, webmail etc. Anything service originating from the blacknight.com domain name is affected.

Update: 11:50: Google have removed the block and access to all services such as Webmail, Exchange Owa and our main site. It will probably take a bit more time for all browsers to pick up the change.

The MySQL Server mysql452.cp.blacknight.com with IP address 81.17.254.3, will be offline for approximately 45 minutes from 9pm Wednesday the 26th of October for scheduled maintenance.

pemlinweb08 is been upgraded tonight 25/10/2011 at 9pm, we expect the upgrade to take up to 2 hours.

Shared hosting server affected:
Pemlinweb08

Update 22:10
Server is back online after upgrade

Summary: It has been 2 years since we last had to update the SSL cert for mail.blacknight.com. Yesterday we applied the new SSL cert on the mail servers. At the time we didn't include the CA bundle. However this morning we've added the CA bundle so this should mean less errors in mail clients.

If you have a mail client that is throwing an error because the cert changed please tell your mail client (outlook, thunderbird, apple mail etc) to accept the new certificate.

PEMVZWIN16 requires a reboot for some updates. The server will be rebooted at 7:45am Tueday the 25th of October.

VPS' Affected:
78.153.196.96       
78.153.196.141     
78.153.196.237     
78.153.196.248            
78.153.196.249      
78.153.196.251   
78.153.196.254      
78.153.197.1
78.153.197.5       
78.153.197.9        
78.153.197.12       
78.153.197.10  
78.153.197.13       
78.153.197.2      
78.153.197.14 
78.153.197.15      
78.153.197.16   
78.153.197.17     
78.153.197.18      
78.153.197.20     
78.153.197.21   
78.153.197.30   
78.153.197.23   

Update 08:00
Server has rebooted and VPS' are starting

The above Linux VPS Server seems to be having difficulty this morning. We're working on it now to get it back. This has a number of VPS on it and their primary IPs are: We hope to have it back up and running asap.

Summary: At 23:00 tonight Friday 21st of October we're going to update the NIC card drivers for the 10GigE network cards that provide connectivity to the SAN. To do this we have to take mail down for a window of approx 15 minutes.

This is to hopefully resolve a bug that we've seen manifest a number of times since the mail storage platform was upgraded last Friday night. Once this is complete we hope that the mail system will function better moving forward.

Summary: 2 of the 4 servers that deliver mail to mailboxes currently have a few thousand messages in their queues at the moment. We expect this to die down in the coming 60-90 minutes as it was due to a number of large mailers from various companies hosted outside the qmail cluster. As a result the spam scanning servers are very busy.

Additionally we're moving the primary spam scanning box to new hardware which should improve performance 10 fold and increase delivery times in periods of high inbound email load.

Update:
The upgrade has greatly improved performance.

The server pemvzlin13 needs to be rebooted for emergency maintenance purposes.  The server will be taken offline to perform a standard file system check and then rebooted.

The maintenance window will start at 07:00 tomorrow morning (20/10/2011) and is expected to take between thirty minutes to two hours, depending on the speed of the file system check.

The following IP addresses will be affected:

78.153.208.24
78.153.208.55
78.153.208.61
78.153.208.146
78.153.208.183
78.153.208.207
78.153.208.227
78.153.208.253
78.153.209.35
78.153.209.101
78.153.209.104
78.153.209.168
78.153.209.171
78.153.209.186
78.153.209.216
78.153.209.220
78.153.209.221
78.153.209.236
78.153.210.13
78.153.210.19
78.153.210.59
78.153.210.102
78.153.210.131
78.153.210.167
78.153.210.185
78.153.210.208
78.153.210.252
78.153.211.13
78.153.211.15
78.153.211.22
78.153.211.26
78.153.211.36
78.153.211.45
78.153.211.50
78.153.211.54
78.153.211.58
78.153.211.124
78.153.211.129
78.153.211.155
78.153.211.157
78.153.211.161
78.153.211.172
78.153.211.176
78.153.211.177

Update 8:00

Maintenance completed

Summary: pemvzlin02 seems to be having difficulty this morning. We're working on it now to get it back. This has a number of VPS on it and their primary IPs are:We hope to have it back up and running within the next 15 minutes

Update 9:10
VPS servers are now coming back online

pemlinweb71 & pemlinweb72 will be offline this evening the 17th October for approximately 45 minutes from 9pm to allow for migration to hardware.

Servers Affected:
pemlinweb71 78.153.215.164
pemlinweb72 78.153.215.165

Update 21:40
The migration is going slower than expected but it's 100% required to ensure continuity of service, we will update as soon as we have more information.

Update 22:00
pemlinweb72 78.153.215.165 has been migrated and booted. pemlinweb71 is ongoing.

Update 22:20
Both servers are now migrated and online.

We're currently experiencing issues with our qmail cluster, mail.blacknight.com - Our engineers are working to resolve this ASAP and all updates will be posted here as they become available.

pemlinweb71 & pemlinweb72 are currently offline we are investigating the issue and will update.

Servers Affected:
pemlinweb71 78.153.215.164
pemlinweb72 78.153.215.165

Summary: Following on from this weeks earlier maintenance we're going to do the final move of e-mail to the new storage platform tonight October 14th.

We'll take mail offline again at 23:00 hours for approx 2 hours until 1am. This move is essentially just an rsync of data from the old system to the new system and that's it. We've already seeded the data to the new platform using a recent backup so there isn't a huge amount of data to be copied. We will then re-mount the new drives onto the mail servers and begin the mail flow once again after some essential tests.

Tomorrow Saturday 15th we'll monitor everything closely and see how it all performs.

pemlinweb71 & pemlinweb72 are currently offline we are investigating the issue and will update.

Servers Affected:
pemlinweb71 78.153.215.164
pemlinweb72 78.153.215.165

Update 8:50
Servers are back online after a reboot

Summary: pemvzlin02 seems to be having difficulty this morning. We're working on it now to get it back. This has a number of VPS on it and their primary IPs are:

Summary: Since approx 11:30 this morning people have been experiencing some delays in downloading e-mail, it appears to download very slow and sometimes it times out. Between 11:30 and approx 13:30 is the busiest time of day for the mail servers and we expect that a rebuilding raid array may be causing this issue. The array should be rebuilt completely by mid to late afternoon today at which time things should improve dramatically. We can't really do anything to speed this up in the interim.

Additionally once we perform the major upgrade later this week of the storage backend we expect mail to function 100% and perform much much better than it has been.

Update @ 16:45:  Mail raid rebuild is still continuing at this time.

Summary: These 3 virtual machines are in the bottom of the rack where the mail equipment was being moved from. The power cable got snagged when we were removing the equipment from the rack.

pemlinweb91 & pemlinweb92 are currently offline, we are investigating the issue and will update.

Servers Affected:
pemlinweb91 78.153.215.218
pemlinweb92 78.153.215.219

Update 16:15

Node has been rebooted and both servers are back online.

Summary: On 23:00 on Monday 10th of October we're going to take the mail cluster offline completely for approx 4 hours until 03:00. This is to facilitate the following:

1) Upgraded storage space for the mail store
2) Upgraded performance for the mail store
3) Upgraded file system from ext3 to ext4 for performance reasons

We're doing this in stages.

Stage 1)

We've restored the existing mail system from a recent backup to a new location. We'll begin copying that data to the new Mail Storage node which has 20Gbit/s of connectivity to our SAN. We estimate that this will take approx 24-48 hours to complete.

Stage 2)

We will then take the mail system offline again in a couple of nights to do a final sync of the changes on the new system. This will depend largely on the amount of time it takes to do the copy over to the SAN. Estimations range from between 36 and 50 hours approx.

Tonight we're simply relocating the servers to a new rack in close proximity to our SAN. And while they're down we're doing a clean up of old mailboxes and purging data that is no longer required.

Summary: pemvzlin02 seems to be having difficulty this morning. We're working on it now to get it back. This has a number of VPS on it and their primary IPs are:

Summary: Parallels have informed us of an update for our billing system which prevents a number of serious problems from re-occuring. Namely it'll prevent more than 1 daily billing process from running. After the upgrade from PBA 5.0 to 5.1 we had a situation where upto 3 daily billing processes were able to run. This resulted in around 150 subscriptions being double billed. This update fixes this problem.

Between 03:00 and 06:00 the store will be up and down, also the billing tab in the control panel will be down for much of it.

This server is currently experiencing difficulties serving HTTP traffic.

 

We are aware of the issue, and hope to have a resolution as soon as possible.

Update 17:30

Issue is now resolved

In an effort to keep all shared hosting environments up to date with the most recent kernels we need to reboot the hardware node that both pemlinweb15 and pemlinweb16 live on tonight.

We are currently experiencing issues with our phones in our Carlow offices

UPDATE 13:00 The phones appear to be working properly at present however we are still investigating the issue ie. what caused the problem

UPDATE 1328 we are still looking into the cause of the issue, but rebooting one of the firewalls seems to have fixed it. At present the network in the office and our phonelines appear to be stable so we'll tentatively say that this issue has been resolved

Services on gorlois.blacknight.ie have become non-responsive and the server requires a reboot.

We hope that the server should return to normal after the reboot, however engineers will be monitoring should there be further issues.

Update @ 11:08:

The issue has now been resolved

The MySQL Server mysql452.cp.blacknight.com with IP address 81.17.254.3, will be offline for approximately 30 minutes from 7am Tueday the 4th of October for scheduled maintenance.