Wednesday, July 13, 2016

Making a Mailserver (Part 6) - Server Migration & Backup

This is an instalment in my series on setting up a Linux based mailserver. See these posts:

In this post, I cover the steps you must plan for when migrating your mail services from one server to another. Plan now for a migration before everything is a distant memory. I won't get into deep technical detail here because the technical detail is covered in the previous posts.

Careful Planning Eases the Transition
While first setting up your mailserver, make copious notes for speedier disaster recovery. After setting up a server of any kind, I like to dump the history command to file and save that off-box.

Something to ponder: could you recreate your server from the backups alone?

Backups

If your VPS provider isn't taking care of your backups (this often comes at an additional cost) then you really should backup frequently. If you're a home-brew error-prone control-freak like me, you'll want to custom script the backup process because the job is otherwise tedious and you'll never get around to it.

If you've followed my posts, then you need to backup the following directories completely:
  • /etc/
  • /home/
  • /var/www/
  • /usr/lib/cgi-bin/
  • /var/mail/
The active inbox files are in /var/mail while the user's mail folders will be stored in their /home path. Backing up those can take some time.

Leave no configuration file behind and grab /etc, this directory doesn't take a lot of space.  The /var/www and /usr/lib/cgi-bin/ dirs are just in case you were messing around with custom web pages despite my constant pleas that you don't.

You might also want to collect a list of all installed packages.

Over time I developed a script to take care of this and scp the backups to a remote server. You can easily adapt it for your needs. Grab my backup-server.sh script from github.com if you like.

Warning! If you do use my backup-server.sh script, be aware that using ssh-copy-id is unwise unless you secure your private key with a password when creating it (ie. with ssh-genkey). The reason is that if you are using a cloud based VPS, the Administrators at that VPS provider can clone your server and access any file they please. If your VPS is installed on an encrypted volume, that's great, but a hacker who compromises your running VPS will also have access to your keyfiles.

Hence I strongly recommend you do one of two things:
  1. Put a password on your private ssh key when generating it.
  2. Run the script using a one-off key:
    • Create a passwordless one-off key to copy to the remote server.
    • Run the script.
    • Delete the local keys (in ~/.ssh/) and the public key from the remote server (in ~/.ssh/authorized_keys).
I've been considering scripting option 2.

Why Would I Need to Migrate?

In my experience, migrations are done only because your hand has been forced and the decision is out of your control. To name a few obvious reasons:
  1. You lost your nerve and decided to migrate the data from the clapped-out 486 in your shed to a VPS in the cloud.
  2. You found a better service for half the price.
  3. Your existing VPS provider declared that their system upgrade requires you to migrate to a new VPS. One VPS provider pulled this trick on me twice in two years and it's very annoying.
  4. Your Operating System went End of Software Support. Running a distribution upgrade is nerve-wracking to the point that I prefer to rebuild from scratch from a clean install.
  5. You prefer to operate as the root user and ran "rm -rf /".
  6. Your system has been compromised.

Migration Steps

Server migration involves, these steps in this order:
  1. Prepare the new server by installing all the packages required by your old server. 
  2. Configure the new server. You'll be recycling the config files from the old server to do that. You can recycle the FQDN and keep the same hostname, exercising some caution to not get the two confused.
  3. Make sure that the users from your old server /etc/passwd & /etc/group are inserted into the new server using the same uid and gid otherwise /home and /var/mail/ is going to be a mess. Don't forget to check /etc/aliases.
  4. Ensure the new server doesn't have the mail daemon (postfix/sendmail/exim) started. 
  5. Backup the old server.
  6. Take the old server mail daemon offline (turn off postfix/sendmail/exim).
  7. Update the MX record to point to the new server, but keep the new server's mail daemon offline.
  8. Copy all the mailboxes and user directories over to the new server. It's easier to tar and compress these (preserving file ownerships and permissions) and then copy, than to scp the files in place. You need to copy many things as root while still preserving the file ownerships and permissions.
  9. Start the mail daemon on the new server. 
  10. Fix all the problems.  
  11. Once you're really sure that the new server is working as expected, shut down the old one. Goto 10.
Here's why you want to do those steps in order:
  • You don't want to have two mailservers running at the same time.
  • The DNS changes take hours to propagate, so as long your old mailserver is offline, you might as well do the DNS change straight away.
  • You make the copy of mailboxes and user directories with the knowledge that they cannot change and the version you are copying is the most current.
It's okay for your mail domain to go dark for a few hours. The Internet doesn't really mind. Most mail hosts make a few attempts over several hours before giving you up for dead. Interestingly, I've noticed that Google / Gmail is very quick to notice your MX server change (less than 30 minutes).

Data Security in the Cloud

As I mentioned earlier, unless you are running from an encrypted volume, your data is open to the VPS company's admins. Either you encrypt your volume or you have no offline data security. You need to think about what will happen to your data when you retire your old server instance.

I know of no elegant solution to the problem of protecting the data on an abandoned VPS. The data security policy of a VPS company should inform your choice of provider.

If you're on a cloud VPS, delete all of the sensitive directories on the old server; likely the same directories that you would backup. It seems possible to me that your VPS provider could still recover and steal the data. There's also a chance that once the VPS provider assigns the disk area to another customer then that customer may be able to recover files.

You could delete the sensitive data and then fill the disk to capacity with junk. Redirecting the output of "yes" to file would be a quick way to do it. An ultra paranoid approach would be do that several times.

No comments:

Post a Comment