Hazel Smith gave an excellent talk at FLOSS UK‘s Unconference in London last weekend about Linux Capabilities and using them to run a backup system with minimal permissions. Several people in the room, myself among them, sat up and went “Nice idea. I’ll be using that”. Here’s what I did:
For many years, I’ve used rdiff-backupÂ to back up a bunch of different Linux systems. It works well, keeps the most recent backup on disk in its original form (including file owners and permissions) and allows access to the previous n days’ worth of backups too, stored efficiently. I keep 30 days and it’s occasionally been handy to have the history available.
My system was “pull” based as you’d expect: the backup server logged into each of the to-be-backed-up systems over SSH and ran rdiff-backup via sudo. You can then configure sudo so that the “backuphelper” user rdiff-backup logs in as can run rdiff-backup as root without a password being prompted for. This then gives rdiff-backup the power to read all the files and do a system-wide backup. On the receiving end, the entire rdiff-backup process and the scripts calling it run as a cron job under the root user so it can preserve owners and files on the backed-up data.
This system had served me well for a number of years, but as Hazel’s talk pointed out, it definitely violates the principle of least privilege to run the entire process as root on both the backup and target servers.
Fixing it: the target servers
I followed a similar process to the slides. On Debian, if you haven’t fiddled with the contents of /etc/pam.d then the installation of the libpam-cap package automatically adds the necessary line for pam_cap.so to common-auth, so it works for SSH, cron and su spawned shells and you only need to configure in /etc/security/capabilities.conf.
Because rdiff-backup is written in Python, you can’t set capabilities on the script itself: the shell spawns a Python process so you need to set the capabilities on /usr/bin/python (or rather /usr/bin/python2.7 at the time of writing, as the former is a symlink). There is much waffle on the internet about how unfortunate it feels to be putting capabilities on the interpreter rather than the intended program. However, since in this case the capabilities only work if Python is run from a user who has already been granted them by pam_cap, it doesn’t seem like too much of a problem. I briefly pondered using a hard link python-with-extra-caps which was owned by ‘backuphelper’, but that felt like a maintenance burden. Overall, I still think doing it this way exposes a much smaller attack surface than running rdiff-backup as root.
Taking it further
I see that Hazel’s post-talk additions to the slides noted the further possibility of using capabilities to avoid running the backup process as root on the backup server. I had a go and managed to get it working.
Again, because rdiff-backup is Python, the capabilities needed setting on the Python interpreter. Once more I made a dedicated ‘backuphelper’ or similar user to run the backup process. I moved the SSH keys and known_hosts list across from ‘root’ which used to own them, and found that rdiff-backup at the backup server end needs CAP_DAC_OVERRIDE (the ability to write arbitrary files, not just read them), and also CAP_FOWNER if you’re expecting it to preserve Linux owner/group information and permissions.
I don’t think the backup server has gained a great deal of security from this: if you can write and chown/chmod arbitrary files, then you can certainly take total control of a system in a few steps. But at the very least, not running it all as root limits the damage that can be done by accidents (bugs/flaws in rdiff-backup) and adds more steps to get in the way of a poorly crafted or targeted exploit.
A couple of other twiddles were needed: if your backup process sends e-mail, it will now do so as ‘backuphelper’ not root, so make sure that user has a sensible from address. Lastly, my backup scripts run ‘shutdown’ to switch off the backup server when it’s finished work. I had to arrange for this to be done via sudo now that the entire backup process is no longer run by root.
As in many configuration files, order matters in the PAM configuration. RTFM. If you want to see whether pam_cap is working, you can do this:
grep CapInh /proc/$$/status
And use capsh –decode= on the resulting bit string to understand what you’ve got. If it’s all zeroes, check capabilities.conf and your PAM configuration.
If you’re thinking of giving your backup system an overhaul, don’t forget to test whether the backups the new system takes can actually be restored from.