Linux VPS node having difficulty - Pemvzlin03

Message

We are currently experiencing some technical difficulties with one of our VPS nodes - pemlinvz03.  We have dispatched an engineer on-site to look in to this right away and we will update this post as soon as we have more information.

The primary IP address of affected VPSs are:

78.153.208.128
78.153.208.130
78.153.208.142
78.153.208.150
78.153.208.164
78.153.208.147
78.153.208.173
78.153.208.182
78.153.208.192
78.153.209.91
78.153.208.195
78.153.209.112
78.153.209.125
78.153.209.137
78.153.209.146
78.153.208.71
78.153.209.14
78.153.209.74
78.153.209.89
78.153.210.129
78.153.210.138
78.153.208.152

Update: 08:45: the node was completely unresponsive so unfortunately we had to reboot it. It's currently doing an fsck on the /vz partition (which is around 700-800GB) which is at 32% approx. We estimate that it'll be back by around 09:30. Sorry for the inconvenience.

Update: 09:20: The disk check had to be restarted, however it's already at 54% so it's going well. ETA now is 09:45

Update: 09:40: The disk check is at 65% currently and going well. We estimate that it'll be between 10:00 and 10:20 when it'll finish.

Update: 09:50: It has restarted from the beginning again, we're keeping a close eye on it. We've a box built and prepared and we can restore the backup from around 2am on the 21st if we have to.

Update: 10:25: It's still going. In the mean time we're doing a bare metal restore of the box to a spare box that we have in place for just such an occasion. We'll know more in the next 30 minutes on what the ETA will be.

Update: 10:50: Ok the file system check finished sucessfully, the machine has rebooted as normal and it's now back online. All the containers are doing disk checks themselves but they're starting slowly. When they're all back online we'll post one further update.

Update: 11:50: All the Containers on this node are now fully back up and running.