Using Duplicity for Full Server Backup on Ubuntu 12.04

duplicity is a great full featured backup tool, providing "encrypted bandwidth-efficient backups using the rsync algorithm." I’m choosing it for my backup needs primarily for the follow reasons:

  • Supports a fleet of backup server types, and even a chroot’d SFTP server will work
  • Works with Linux, FreeBSD, and OpenBSD, out of the box
  • Does not require root access on the backup server

To perform full server backups using duplicity on your Ubuntu 12.04 system, simply perform the following.

  • Install duplicity:

    sudo apt-get install duplicity
    
  • Perform backup (using rsync method):

    $ sudo -H duplicity --no-encryption \
      --exclude /mnt --exclude /tmp --exclude /proc --exclude /sys \
      / rsync://<user>@<host>//<backup-directory>
    

    You’ll need to replace <user>, <host> and <backup-directory> with your backup username, host, and directory, respectively. See the "Full Example" below for more info.

    Output should look something like:

    Import of duplicity.backends.sshbackend Failed: No module named paramiko
    Import of duplicity.backends.giobackend Failed: No module named gio
    Local and Remote metadata are synchronized, no sync needed.
    Last full backup date: none
    No signatures found, switching to full backup.
    

    If you’d like your backups to be encrypted, simply remove the –no-encryption option and provide a passphrase when prompted. You’ll need this same passphrase to restore from a backup. It is a symmetric key. Keep the key very safe; if you lose it, your backups will be useless.

  • Schedule regular backups

    The next step is to set this up in cron, so your backups are done regularly. The following entries in /etc/crontab will perform a full backup once a week, Sunday at midnight, then incremental backups the other 6 days of the week (also at midnight). Additionally, we want to clean up backups that are too old, so we add an additional entry to keep only the latest two full backups (plus incrementals).

    0 0 * * sun     root    duplicity full --no-encryption --exclude /mnt --exclude /tmp --exclude /proc --exclude /sys / rsync://<user>@<host>//<backup-directory>
    0 0 * * 1-6     root    duplicity --no-encryption --exclude /mnt --exclude /tmp --exclude /proc --exclude /sys / rsync://<user>@<host>//<backup-directory>
    0 8 * * sun     root    duplicity remove-all-but-n-full 2 --force rsync://<user>@<host>//<backup-directory>
    

Full Example for an ARP Networks VPS

At ARP Networks, we provide raw backup space to customers for 10 cents per GB. The example below will illustrate how to do a full backup of your Ubuntu or Debian VPS. Assuming your backup username is johndoe, simply perform the following steps.

  • Add root SSH public key to backup server

    Create an SSH key for root as follows:

    sudo -H ssh-keygen -t rsa -b 2048
    

    Accept the defaults and choose a good passphrase.

    See the Portal, click on the Backup Space service, and then click the link to submit an SSH key.

  • Install duplicity:

    sudo apt-get install duplicity
    
  • Perform backup (using rsync method):

    $ sudo -H duplicity --no-encryption \
      --exclude /mnt --exclude /tmp --exclude /proc --exclude /sys \
      / rsync://johndoe@backup01.cust.arpnetworks.com//home/johndoe
    

    backup01.cust.arpnetworks.com is one of our backup servers. If the Portal lists a different server for you, use that one.

    Additionally, as mentioned above, if you want your backups to be encrypted, remove the –no-encryption option.

  • Schedule regular backups

    Put the following in your /etc/crontab

    0 0 * * sun     root    duplicity full --no-encryption --exclude /mnt --exclude /tmp --exclude /proc --exclude /sys / rsync://johndoe@backup01.cust.arpnetworks.com//home/johndoe
    0 0 * * 1-6     root    duplicity --no-encryption --exclude /mnt --exclude /tmp --exclude /proc --exclude /sys / rsync://johndoe@backup01.cust.arpnetworks.com//home/johndoe
    0 8 * * sun     root    duplicity remove-all-but-n-full 2 --force rsync://johndoe@backup01.cust.arpnetworks.com//home/johndoe
    

    Note: Please adjust the backup time to your own preference, otherwise many backups may fire off at the exact same time, slowing down the host.

…and that’s all there is to it!

How to Add a New Partition on OpenBSD

Following the same theme as my last post, instead of claiming space added to the end of a volume by extending an existing partition (technically, a "slice" in OpenBSD), I will show how to add a new partition using this space.

I find it convenient to create the new slice as the "d" slice and then put /home onto it.

So, let’s get started.

  • Boot into the system’s rescue disk:

    > boot bsd.rd
    

    Then select (S)hell at the prompt.

  • Claim extra space

    Re-initialize the MBR, using the entire disk.

    # fdisk -i wd0
    Do you wish to write new MBR and partition table? [n] y
    Writing MBR at offset 0.
    #
    
  • Add a new slice for /home (the "d" slice)

    This system has only the root slice and swap. We will add a new slice, using all the unused space at the end. In the following example, I have a 120GB volume, where 20GB is used by root and swap, and the rest unallocated.

    # disklabel -E wd0
    Label editor (enter '?' for help at any prompt)
    > p
    OpenBSD area: 64-251658225; size: 251658161; free: 209728575
    #                size           offset  fstype [fsize bsize  cpg]
      a:         38812976               64  4.2BSD   2048 16384 38128
      b:          3116610         38813040    swap
      c:        251658240                0  unused
    > a
    partition: [d]
    offset: [41929650]
    size: [209728575]
    FS type: [4.2BSD]
    Rounding offset to bsize (32 sectors): 41929664
    Rounding size to bsize (32 sectors): 209728544
    > p
    OpenBSD area: 64-251658225; size: 251658161; free: 31
    #                size           offset  fstype [fsize bsize  cpg]
      a:         38812976               64  4.2BSD   2048 16384 38128
      b:          3116610         38813040    swap
      c:        251658240                0  unused
      d:        209728544         41929664  4.2BSD   2048 16384    1
    > w
    > q
    No label changes.
    #
    
  • Create a filesystem on the new slice:

    # newfs /dev/rwd0d
    /dev/rwd0d: 102406.5MB in 209728544 sectors of 512 bytes
    506 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each
    super-block backups (for fsck -b #) at:
     32, 414688, 829344, 1244000, 1658656, 2073312, 2487968, 2902624, 3317280,
     3731936, 4146592, 4561248, 4975904, 5390560, 5805216, 6219872, 6634528,
     7049184, 7463840, 7878496, 8293152, 8707808, 9122464, 9537120, 9951776,
     ...
     [snip]
     ...
     193229728, 193644384, 194059040, 194473696, 194888352, 195303008, 195717664,
     196132320, 196546976, 196961632, 197376288, 197790944, 198205600, 198620256,
     199034912, 199449568, 199864224, 200278880, 200693536, 201108192, 201522848,
     201937504, 202352160, 202766816, 203181472, 203596128, 204010784, 204425440,
     204840096, 205254752, 205669408, 206084064, 206498720, 206913376, 207328032,
     207742688, 208157344, 208572000, 208986656, 209401312,
    #
    
  • Migrate /home

    Here, we mount the root filesystem (which contains the old /home), and also the new filesystem, and finally migrate the data from old /home to new space.

    # mount /dev/wd0a /mnt
    # mount /dev/wd0d /mnt2
    # (cd /mnt/home; tar cf - .) | (cd /mnt2; tar xpf -)
    # rm -rf /mnt/home
    # mkdir /mnt/home
    

    Note, the tar command above is a nice platform independent way of copying everything from one directory to another, preserving everything (permissions, etc…). The cp command is different on Linux and the *BSDs, yet tar, used in the above fasion, is identical on Linux, FreeBSD, NetBSD, and OpenBSD. I thank Todd Fries of Free Daemon Consulting for teaching me that one.

  • Mount on boot

    The following command will add the new /home to your /etc/fstab so it is automatically mounted upon boot.

    # echo "/dev/wd0d /home ffs rw,softdep 0 1" >> /mnt/etc/fstab
    
  • Clean up:

    # umount /mnt*
    # reboot
    

    Now enjoy the extra space!

How to Resize an OpenBSD Root Partition

At ARP Networks, sometimes we have to resize an OpenBSD root partition to satsify a customer requirement, template requirement, or something else along those lines.

I’ve sometimes seen customers struggle to do this themselves, so the following is an example of how I resized a root partition from 5 GB to 9 GB, leaving 1 GB for swap. The underlying volume was already expanded from 5 GB to 10 GB, to give us room to do this.

So, let’s get to it.

  • Boot into the system’s rescue disk:

    > boot bsd.rd
    

    Then select (S)hell at the prompt.

  • Resize:

    Grab the growfs(8) program from the root partition. This isn’t included with the rescue disk, for some reason.

    # mount /dev/wd0a /mnt
    # cp /mnt/sbin/growfs .
    # umount /mnt
    

    Re-initialize the MBR, using the entire disk.

    # fdisk -i wd0
    Do you wish to write new MBR and partition table? [n] y
    Writing MBR at offset 0.
    #
    

    Change the size of the slices.

    This system has only the root partition and swap. The idea is to remove the swap partition, then extend root, and finally put the swap partition back.

    # disklabel -E wd0
    # Inside MBR partition 3: type A6 start 63 size 20964762
    Treating sectors 63-20964825 as the OpenBSD portion of the disk.
    You can use the 'b' command to change this.
    
    Initial label editor (enter '?' for help at any prompt)
    > p
    OpenBSD area: 63-20964825; size: 20964762; free: 9446220
    #                size           offset  fstype [fsize bsize  cpg]
      a:         10490382               63  4.2BSD   2048 16384    1
      b:          1028160         10490445    swap
      c:         20971520                0  unused
    > d
    partition to delete: [] b
    > p
    OpenBSD area: 63-20964825; size: 20964762; free: 10474380
    #                size           offset  fstype [fsize bsize  cpg]
      a:         10490382               63  4.2BSD   2048 16384    1
      c:         20971520                0  unused
    > c
    partition to change size: [] a
    Partition a is currently 10490382 sectors in size, and can have a maximum
    size of 20964762 sectors.
    size: [10490382] 9G
    Rounding to cylinder: 18876312
    > a b
    offset: [18876375]
    size: [2088450]
    FS type: [swap]
    > p
    OpenBSD area: 63-20964825; size: 20964762; free: 0
    #                size           offset  fstype [fsize bsize  cpg]
      a:         18876312               63  4.2BSD   2048 16384    1
      b:          2088450         18876375    swap
      c:         20971520                0  unused
    > w
    > q
    No label changes.
    #
    

    Grow the filesystem using growfs(8)

    # growfs /dev/wd0a
    We strongly recommend you to make a backup before growing the Filesystem
    
     Did you backup your data (Yes/No) ? Yes
    new file systemsize is: 4719078 frags
    Warning: 216792 sector(s) cannot be allocated.
    growfs: 9111.1MB (18659520 sectors) block size 16384, fragment size 2048
            using 45 cylinder groups of 202.47MB, 12958 blks, 25984 inodes.
    super-block backups (for fsck -b #) at:
     10781088, 11195744, 11610400, 12025056, 12439712, 12854368, 13269024, 13683680, 14098336, 14512992, 14927648, 15342304, 15756960, 16171616, 16586272, 17000928, 17415584, 17830240,
     18244896
    #
    

    Give it a rinse.

    # fsck /dev/wd0a
    ** /dev/rwd0a
    ** Last Mounted on /
    ** Phase 1 - Check Blocks and Sizes
    ** Phase 2 - Check Pathnames
    ** Phase 3 - Check Connectivity
    ** Phase 4 - Check Reference Counts
    ** Phase 5 - Check Cyl groups
    100205 files, 418854 used, 4172217 free (97 frags, 521515 blocks, 0.0% fragmentation)
    
    MARK FILE SYSTEM CLEAN? [Fyn?] y
    
    
    ***** FILE SYSTEM WAS MODIFIED *****
    #
    

    And that’s pretty much it. Everything is done within OpenBSD’s rescue disk kernel bsd.rd, no need to have a separate CD-ROM / USB rescue disk utility.

Having a Javascript Moment With Ruby, WTF Is Going on Here…

WTF is going on here?

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
(rdb:1) sum
244.4
(rdb:1) amount
244.4
(rdb:1) sum != amount
true
(rdb:1) sum.to_f
244.4
(rdb:1) amount.to_f
244.4
(rdb:1) sum.to_f != amount.to_f
true
(rdb:1) sum == amount
false
(rdb:1) sum.to_f == amount.to_f
false
(rdb:1)

First guess is some type conversion problem…

Still investigating.

Using the Quota Command Within a Chroot Environment

I was recently using Jailkit on Ubuntu to set up a chroot environment for users of a backup storage server. I wanted to give the users the ability to check their quotas with the regular quota command.

I go about telling Jailkit that the chroot directory will need the quota binary. This should be all that is needed, or so I thought.

Upon checking a quota as a chroot’d user, it barfs the following error,

1
2
3
4
$ quota
quota: Cannot open quotafile /home/aquota.user: Permission denied
quota: Quota file not found or has wrong format.
$

From what I can tell, this is a very misleading error message. The permissions of aquota.user are always 0600, readable / writable only by root. This can’t be changed, not even as root. Witness.

1
-rw------- 1 root root 7168 2013-02-27 22:32 /home/bkusers/home/aquota.user

Using a bit of strace and some luck, I find what the quota command is really looking for is /proc/sys/fs/quota.

It simply does a stat() system call on that directory within proc, so if you simply do,

1
$ sudo mkdir -p /home/bkusers/proc/sys/fs/quota

then the quota command works like a charm within the chroot environment.

1
2
3
4
5
$ quota -s
Disk quotas for user testuser (uid 1007):
     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
       /dev/md3      16       0   3907M               4       0       0
$

My chroot directory in these examples was obviously /home/bkusers, modify accordingly for your needs.

The ‘Yes’ Men

Let me turn a corner and cover a non-technical topic here for once…

Several years ago, I had a job where I had to ask myself the following question, “Do I do what I think is right, or do I do what will make my new boss happy?”

Now, this was not a moral question, it was simply about how to do some measurements / benchmarking and systems stuff; pretty low key, no big deal.

I stuck to my guns and at some point shortly after, I didn’t have that job anymore. ;)

In my case, there really wasn’t any big life altering changes to myself or anyone else regardless of which decision I made. But in some cases, this can really go the wrong way…

Let me illustrate by quoting something from my reading last night; the book was Amy Goodman’s (Host of Democracy Now!) ”Breaking the Sound Barrier”, and within was a quote from Grammy Award-winning soul singer John Legend when he gave the commencement address at the University of Pennsylvania,

Too often, in business and in government, people are rewarded for having the answer that the person they report to wants them to have: “Yes, sir. We can provide mortgages to people who have no down payment and can’t afford the monthly payments.”… “Yes, ma’am. I can write a legal brief to justify torture.”

John Legend Commencement Address at the University of Pennsylvania 2009

Stick to your guns, fuck your boss.

Migrated From Mephisto to Octopress

I’m proud to say I’ve dusted off this blog and migrated from Mephisto to Octopress. Now I can actually post new content (the old Mephisto system got so dated and went to dependency hell that I was hardly able to do anything besides keep the static cached files up).

I’m going to try to go through all the posts and cleanup / fix any loose ends. So far the import went pretty smooth and I’ve only burned a few hours (instead of a whole weekend).

I was unable to successfully import the comments, so those will be gone forever. Most were spammy anyway, and any comments posted at least within the last year or two use Disqus, and those are still around.

I’d like to get my usual theme working again, but alas, that is very low priority right now. Perhaps a new look is warranted anyway. However, at least, I can move away from the default theme. Gotta read more docs…

Let me know if anything is out of the ordinary.

Thanks!

Running OpenBSD 4.5 in KVM on Ubuntu Linux 9.04

Here’s a quick post on how to get OpenBSD 4.5 working as a KVM guest on kvm-84, the version that ships with Jaunty.

Unlike OpenBSD 4.4, which worked out of the box, 4.5 made some changes in their kernel with respect to interrupts that may have exposed bugs in KVM/QEMU. The workaround is to disable mpbios within the OpenBSD kernel. This will affect SMP guests adversely, so it isn’t for everyone.

Your kernel is affected by this if you boot and it hangs at:

1
setting tty flags

Credit goes to Todd Fries of Free Daemon Consulting, LLC for figuring this out and passing down the wisdom onto me.

Note: I’m using a 64-bit host as well as the 64-bit version of OpenBSD as the guest.

Step 1

Boot the OpenBSD 4.5 install CD within your blank VM and proceed through the entire install normally. Then, reboot from the hard disk but stop at the boot prompt

Step 2

At the boot prompt, perform the following:

1
2
3
4
5
6
boot> bsd -c
...
UKC> disable mpbios
 54 mpbios0 disabled
UKC> quit
...

System will now boot normally.

This has temporarily disabled mpbios and will not persist across reboots. You now have to make this change permanent in the OpenBSD kernel, as illustrated in the next step.

Step 3

Login to your new system as root and perform:

1
2
3
4
5
6
7
8
9
# config -ef /bsd
OpenBSD 4.5 (GENERIC) #2052: Sat Feb 28 14:55:24 MST 2009
    deraadt@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC
Enter 'help' for information
ukc> disable mpbios
 54 mpbios0 disabled
ukc> quit
Saving modified kernel.
#

Now your system will boot up normally from this point onward.

New Server

Moved this blog to a new and faster server. Maybe now I’ll start posting again, since the admin section isn’t slowed to a crawl any more with spam comments. :(

If anyone notices anything missing, blow up, etc… Please leave a comment. Thanks!