Blacknight Hosting Network Status

Payments processer Realex payments issues.

2012-05-18T10:34:24Z

The payments processor Realex is currently having some technical issues which means we can't accept any credit card or laser card payments for the moment - Paypal is still an option though and customers can also use Electronic funds transfer (EFT).

pemvzmps28 reboot

2012-05-16T23:18:14Z

The server pemvzmps28 will be rebooted at 8am Thursday 17th May, to perform maintenance tasks.

This will affect the following pemlinweb servers + IP's:

pemlinweb25 + pemlinweb26

81.17.255.23
81.17.254.67
81.17.255.27
81.17.255.94
81.17.255.22
81.17.255.126
81.17.255.20
81.17.255.21
81.17.255.104
81.17.254.64
81.17.255.26
81.17.255.51

We estimate this should take no more than 10 minutes.

Scheduled maintenance upgrade of our Cloud platform Tuesday 15th May

2012-05-15T08:48:29Z

We are upgrading our OnApp software to a new version and will be performing the upgrade on today Tuesday 15th May 2012 at 10:00.
We expect the following to be affected /offline for a period of 2 hours approximately (from 10:00 am)

Services affected : the cloud control panel , and cloudstore portal will be offline and customers will be unable to upgrade their subscriptions until the update has completed.

We would request any users to defer upgrading/altering their subscription until after the upgrade.

Update 11:50: The upgrade is currently underway the cloudstore is offline. We will update this post once it completes.

Update 1:00pm: The cloudstore is now back online, this upgrade will be completed tomorrow morning at 10:00am when the store will go offline again.

pemvzmps28 drive replacement - 8am Tues 15th May

2012-05-14T10:18:08Z

The server pemvzmps28 will be taken offline at 8am tomorrow morning in order to perform a drive replacement.

This will affect the following pemlinweb servers + IP's:

pemlinweb25 + pemlinweb26

81.17.255.23
81.17.254.67
81.17.255.27
81.17.255.94
81.17.255.22
81.17.255.126
81.17.255.20
81.17.255.21
81.17.255.104
81.17.254.64
81.17.255.26
81.17.255.51

We estimate this should take no more than ~30 minutes.

UPDATE 08:37 - This is now complete

MyLittleAdmin upgrade

2012-05-11T13:40:23Z

We are currently upgrading the version of MyLittleAdmin available in your Control Panels for Microsoft SQL servers. MyLittleAdmin will be unavailable for a period of approximately 30 minutes from 15:00 to allow us to carry out this upgrade.

pemvzmps68 down

2012-05-11T07:24:30Z

Our shared hosting node Pemvzmps68 is currently experiencing issues - affected services are

pemlinweb69 / 70 the following ip's are affaected

78.153.214.200
78.153.214.206
78.153.214.226
78.153.214.214
78.153.214.197
78.153.214.209
78.153.214.232
172.16.4.216
78.153.215.162
78.153.214.245
78.153.214.153
78.153.215.135
78.153.214.196
78.153.214.217
78.153.215.140
78.153.214.208
172.16.4.217
78.153.215.163
78.153.214.84
78.153.214.179
78.153.214.188
78.153.215.31
78.153.214.128
78.153.215.32
78.153.215.81
78.153.215.129
78.153.215.138

We are investigating the the cause and will update shortly

Update 8:47 : we have replaced a faulty component in the server and all services have resumed

Scheduled maintenance upgrade of our Cloud platform Tuesday 15th May

2012-05-09T15:30:16Z

We are upgrading our OnApp software to a new version and will be performing the upgrade on Tuesday 15th May 2012 at 10:00.
We expect the following to be affected /offline for a period of 2 hours approximately (from 10:00 am)
Services affected : the cloud control panel , and cloudstore portal will be offline and customers will be unable to upgrade their subscriptions untl the update has completed.

We would request any users to defer upgrading/altering their subscription until after the upgrade.

Update: this remains scheduled for 10:00 Am this morning

Shared Linux server problems

2012-05-08T18:52:51Z

We are currently experiencing issues with one of the hardware nodes serving Linux Shared Hosting. We have dispatched someone to take a look at the server, and hope to have the issue resolved as soon as possible.

Services affected: Linux Shared Hosting on the following IP addresses

Pemlinweb101
78.153.215.244
78.153.215.109
78.153.215.118
78.153.215.121
78.153.215.123
78.153.215.126
78.153.215.133
78.153.215.136
78.153.215.142
78.153.215.147

Pemlinweb102
78.153.215.137
78.153.215.131
78.153.215.245

PemSaaS01 (Go-Mobi / Mx Sweep )
78.153.215.249
78.153.215.120
78.153.215.122

Update 22:15 - These issues have now been resolved.

Hosted Exchange Public Folders

2012-05-08T16:09:12Z

We are currently experiencing an issue with some customer Public Folders, which may make them inaccessible for certain clients. We hope to have the issue resolved as soon as possible.

Update: 17:30 - These issues have now been resolved.

Backnet issues - shared hosting

2012-05-08T08:42:06Z

We are currently experiencing issues with some shared servers being unable to reach other internal servers, including MySQL and mail.

We are investigating and will update as soon as we have further information.

Update @ 09:55: The issue has been identified and resolved.

Scheduled Maintenance on Cloudstore : Thursday 3/5/12 14:30 - 16:00

2012-05-02T16:02:30Z

We are scheduling a maintenance window to move part of the Cloudstore software to a new hardware node. We estimate this will take a 1.5 hours to move and complete testing.
Customers may be unable to purchase templates from the cloudstore during the duration of this outage. There will be no disruption to the operation of existing Vm's.

Update : 16:45 - All works have been completed

PEMLINWEB36 And PEMLINWEB37 Issues

2012-04-30T23:00:14Z

pemlinweb36 and pemlinweb37 are currently unresponsive. The issue is currently being investigated.

UPDATE 00:15: Both servers are back up and running.

pemlinweb36:
81.17.254.46
81.17.255.59
81.17.255.63
81.17.255.114
81.17.255.127
81.17.255.131

pemlinweb37:
81.17.254.20
81.17.255.89
81.17.255.111
81.17.255.115
81.17.255.34
81.17.255.102

mail.blacknight.com / smtpr1.cp.blacknight.com issues

2012-04-25T10:52:39Z

We are expierencing issues at the moment with the shared mail system, we are working to bring the system back online shortly.

We will update this status post again once we have more information.

Services Affected: all webmail, pop, imap, smtp services for people using qmail shared hosting are currently down.

Update: 12:30

The mail storage system experienced a kernel panic around 11:30am this morning. As a result imap, pop3, smtp, webmail etc went down at this time. We had to initiate a reboot of the Mail Storage system in order to recover from this and it's currently doing a disk check on the partition that contains the e-mail.

The partition check will take some time to complete unfortunately, approximately 1 hour and 30 minutes based on past experience.

Update: 12:40

As people are asking our helpdesk certain questions we'll try to address there here,

Mail being delivered to you won't be lost. For the most part the majority of email coming into us comes in through one of our relay servers. These servers will queue the email until Qmail is fully back up and running.

Update: 14:00

The disk check was progressing at 5% every 8-10 minutes but it has slowed down dramatically and it's currently at approx 40%. We are readjusting our estimate on the completion of the disk check to be approx 16:00 or sooner hopefully.

Apologies for the delay but this _has_ to be done, especially after a kernel panic..

Update: 16:15

Services are now coming back online, pop, imap and smtp services are resumed, we expect to have webmail back online at approx 16:50.

Update 17:00

We are seeing a huge demand on the servers since we brought them back online due to the number of connections and the backlog of email. We are stopping inbound email until 18:00 to allow the servers to deal with the backlog. This will also delay the start of webmail again which we now expect to bring back online approx 18:00

Update 1900

Apologies for the delay in posting an update.

All mail services (POP, SMTP, IMAP and Webmail) on the Qmail system are fully operational since 1815.

Shared MySQL server issues

2012-04-24T23:48:28Z

We are currently experiencing difficulties with one of our shared MySQL servers, we have dispatched someone to reboot it. We hope to have the server back online as soon as possible.

Update 03:55 - The server is now back online after a hardware card change.

Scheduled Maintenance on network equipment in Telecity (DEG) Data Centre

2012-04-24T12:38:23Z

Our engineers will be performing some essential maintenance on our core network routers and switches in the Telecity data centre (DEG) on Saturday 28th April. We are scheduling a maintenance window of 2 hours from 13:00 to 15:00 to complete the reboots required on all the devices. During this period we expect there to be some minor disruption(s) to connectivity whilst the redundant routes re-converge, or whilst rebooting a particular top of rack switch. We expect each "outage" to be no more than 3 minutes in duration. Affected customers will have received an email correspondence with regard to this, which outlined our plans for pre-empting the Telecity power outage scheduled for May12th. Update : All works have been completed and network switches and routers have been moved to redundant power. ]]>