Notification Type
Emergency Maintenance
Date
October 14, 2011 11:00 PM
Service Affecting
Yes
Message
Summary: Following on from this weeks earlier maintenance we're going to do the final move of e-mail to the new storage platform tonight October 14th.We'll take mail offline again at 23:00 hours for approx 2 hours until 1am. This move is essentially just an rsync of data from the old system to the new system and that's it. We've already seeded the data to the new platform using a recent backup so there isn't a huge amount of data to be copied. We will then re-mount the new drives onto the mail servers and begin the mail flow once again after some essential tests.
Tomorrow Saturday 15th we'll monitor everything closely and see how it all performs.
Update: 01:20: We've gone over time on this but the difference between the restored backup and the data on the old mail storage node is more significant than we expected. The copy is approx 50% done and we estimate about another hour to two hours for completion.
Update: 03:30: The copy is going very well, it shouldn't be too much longer. Current estimates for complete are approx 04:15am
Update: 04:15: It's very close to being completed right now. There's probably another 30-40 minutes max left. ETA is now 05:00.
Update: 04:45: Mail has been back and stable for the past 10 minutes. We have full visibility on the storage platforms performance now and everything is well within specified parameters.
Leave a comment