All is not quite well on the Snow Leopard front, however. The rsync that Apple ships (at least the one in 10.6.1) has a bug. When copying a FIFO it attempts to copy the contents of the FIFO. If you know what a FIFO is, you know why this is bad. So rsync hangs in the 90-fifo test of the test suite. When running from a terminal you can work around this by just hitting Control-C, which will cause the copy to stop and the other copiers to be tested. If you want to avoid the hang you can rename tests.d/90-fifo.test to tests.d/90-fifo.test.disabled or whatever floats your boat. I haven’t done that already because, well, a hang is arguably a valid test result!

Oh, and I also fixed a bug with the BSD flags test. If you care about that stuff you should re-test.

One other interesting tidbit — I changed the “lots of metadata” test to lock the file. This trips up some copiers that duplicate the “locked” status before setting up other metadata, causing the other metadata to be lost.

Much to my shame, I had an e-mail upheaval and I can’t find the names of the individuals who reported the BSD flags bug and the “early locking” phenomenon. If that was you, please drop me a message and I’ll give you the credit you deserve.

UPDATE: It was Rob Kennedy who reported the BSD flags bug and the “early locking” phenomenon. I hadn’t thought to search through my blog comments. Silly me! Thanks a lot for those tips, Rob!

Here’s how you test Time Machine with Backup Bouncer:

$ mkdir ~/bb
# We need to trick BB into thinking this is a volume it created
$ touch ~/bb/bbouncer-vol
$ sudo bbouncer create ~/bb
# Now do a Time Machine backup and wait till it's done
$ mv ~/bb ~/bb-orig
# Now restore the ~/bb directory from Time Machine.  
# Try not to catch SPACE MADNESS!!!
$ bbouncer verify -d ~/bb-orig ~/bb

On my Leopard 10.5.3 machine here’s what I get with BB 0.1.3:

Verifying:    basic-permissions ... ok (Critical)
Verifying:           timestamps ... ok (Critical)
Verifying:             symlinks ... ok (Critical)
Verifying:    symlink-ownership ... FAIL 
Verifying:            hardlinks ... FAIL (Important)
Verifying:       resource-forks ... 
   Sub-test:             on files ... ok (Critical)
   Sub-test:  on hardlinked files ... FAIL (Important)
Verifying:         finder-flags ... ok (Critical)
Verifying:         finder-locks ... ok 
Verifying:        creation-date ... ok 
Verifying:            bsd-flags ... FAIL 
Verifying:       extended-attrs ... 
   Sub-test:             on files ... ok (Important)
   Sub-test:       on directories ... ok (Important)
   Sub-test:          on symlinks ... ok 
Verifying: access-control-lists ... 
   Sub-test:             on files ... ok (Important)
   Sub-test:              on dirs ... ok (Important)
Verifying:                 fifo ... FAIL 
Verifying:              devices ... FAIL 
Verifying:          combo-tests ... 
   Sub-test:  xattrs + rsrc forks ... ok 
   Sub-test:     lots of metadata ... ok 

The only annoyance is that hardlinks aren’t preserved, but that’s not surprising considering the way hardlinks are used internally by TM to prevent wasted space. (I should point out that the hardlink failure here is not the same as the Apple rsync one I mentioned earlier today, and it’s not nearly as bad.) These results are not perfect but I would have no problem recommending TM backups to any less-than-hardcore user.

------------------ rsync-apple ------------------
Verifying:    basic-permissions ... ok (Critical)
Verifying:           timestamps ... ok (Critical)
Verifying:             symlinks ... ok (Critical)
Verifying:    symlink-ownership ... ok 
Verifying:            hardlinks ... ok (Important)
Verifying:       resource-forks ... 
   Sub-test:             on files ... ok (Critical)
   Sub-test:  on hardlinked files ... FAIL (Important)
Verifying:         finder-flags ... ok (Critical)
Verifying:         finder-locks ... FAIL 
Verifying:        creation-date ... FAIL 
Verifying:            bsd-flags ... ok 
Verifying:       extended-attrs ... 
   Sub-test:             on files ... ok (Important)
   Sub-test:       on directories ... ok (Important)
   Sub-test:          on symlinks ... FAIL 
Verifying: access-control-lists ... 
   Sub-test:             on files ... ok (Important)
   Sub-test:              on dirs ... ok (Important)
Verifying:                 fifo ... ok 
Verifying:              devices ... ok 
Verifying:          combo-tests ... 
   Sub-test:  xattrs + rsrc forks ... ok 
   Sub-test:     lots of metadata ... ok 

If you add a resource fork to a file that’s hardlinked to another file then only one of the files gets copied. This, no doubt, is due to the hackery involved to make HFS+ look like it has a real hardlinks when it actually doesn’t.

Some of the changes in BB 0.1.3 were a bit invasive and Bash is not the greatest of programming languages, so please keep your eyes open for bugs. If you see something in your results that doesn’t make sense just let me know.

So I wasn’t too surprised when a user forwarded me a somewhat exasperated-sounding note from Shirt Pocket software explaining why SD doesn’t pass BB’s FIFO and device file tests. I won’t paste the note in here, since I feel that would be poor form, but I will pass along their rationale for not copying these files. They feel that FIFOs and device files are normally system-created files that are not meant to be preserved across reboots or backups. For that reason, SD doesn’t back up these filesystem object types at all.

One can understand how Shirt Pocket arrived at this policy. Writing a good system-level backup tool is more subtle than it might appear. There are actually parts of the filesystem that you don’t want to back up, like caches, VM files, and so forth. You don’t want to back up the device files under /dev, since those get created dynamically depending on what devices are attached. In short, there are some parts of the filesystem that are tied to the state of the running system, and these should not normally be copied in a backup.

The problem with Shirt Pocket’s policy is that although it is certainly true that the system-created FIFOs and device files should not be copied in a system-level backup, there are legitimate reasons a user might create FIFOs or device files, and there’s every reason to preserve those user-created objects during a backup. Don’t get me wrong — this is obscure stuff that only advanced UNIX hackers would do, but hey, advanced UNIX hackers need reliable backups too! SD’s policy is, IMHO, overly broad. I would be much more comfortable if SD excluded filesystem locations rather than entire classes of filesystem objects.

The Shirt Pocket letter also accused BB of being “rather misleading to most” on this issue because it makes people worry about something that won’t cause them any problems. Although I vehemently disagree with the “misleading” characterization, I agree that not every BB test is equally important in practical terms. Most end-users will absolutely not care that tool X doesn’t backup FIFOs. This is why BB has always had a prioritization feature. Quoting from the usage string of bbouncer:

-T <set>     Use the given set of tests.  <set> can be either 
             "critical", for tests whose failure will cause problems
             for average users, or "important", for tests whose
             failure may not cause problems for many users, but which
             may be important to power-users or may qualify as 
             critical in the future.

The FIFO/device file tests are not included in either the “critical” or “important” set of tests. So I’m saying (and have always said) that most users, even most power users, won’t care about them. So why test for them at all? Consider the situation before Maurits’ blogged on backups and BB existed. Most people, myself included, didn’t even know what the full set of OS X filesystem object types and metadata was. Maurits filled us in on what existed, but knowing how to test for preservation was still a black art. BB was meant to democratize that testing, but also to act as an exhaustive test set. (I don’t claim to have achieved even close to 100% coverage, but that’s the goal.) So it’s important to me that BB include tests for any metadatum or filesystem object that can conceivably have a reason to be backed up.

Having said that, I’ll concede there’s a usability bug here. The -T option is unlikely to be discovered by casual users, and they’re the ones who need it most! It’s not used by autopilot and it’s not even documented in the README. So I think I’ll be making the following changes:

• I’ll document -T more pervasively
• I’ll have the priority of a test appear in BB’s output
• I’ll have BB default to running in “important” mode, changing the exhaustive mode to an opt-in option

Hopefully this will help prevent confusion for casual users and ease the support burden of tool developers while maintaining BB’s goals of providing easy backup tool validation and exhaustive test coverage.

I just uploaded Backup Bouncer 0.1.2 for your backup-tool-testing pleasure. Get it at the usual place. Not a whole lot has changed, but there’s a fix to extended attribute handling that solves some problems in Leopard. Andreas Fuchs provided a patch for this and also mentioned that SuperDuper can’t handle dangling symlinks! I haven’t verified this, but if it’s true it’s a pretty huge failure for an otherwise excellent tool. In any case, I added a case to the symlink test to check for it.

I’ve also added a check for extended attributes on symlinks. This is pretty obscure, but a well-written tool shouldn’t care, since the extended attribute support would be orthogonal to the file type. If your favorite tool fails that test I wouldn’t worry about it too much, but you might want to file a bug report.

In related news, I read a while back that Apple was using xar for packaging in Leopard. After installing Leopard on my own machine I was a bit disappointed to find that Apple has installed xar 1.4, which does not do a good job of preserving metadata:

------------------ xar ------------------
Verifying:    basic-permissions ... ok
Verifying:           timestamps ...
   Sub-test:    modification time ... ok
ok
Verifying:             symlinks ... ok
Verifying:    symlink-ownership ... FAIL
Verifying:            hardlinks ... ok
Verifying:       resource-forks ... ok
Verifying:         finder-flags ... ok
Verifying:         finder-locks ... FAIL
Verifying:        creation-date ... FAIL
Verifying:            bsd-flags ... ok
Verifying:       extended-attrs ...
   Sub-test:             on files ... FAIL
   Sub-test:       on directories ... FAIL
   Sub-test:          on symlinks ... FAIL
FAIL
Verifying: access-control-lists ...
   Sub-test:             on files ... FAIL
   Sub-test:              on dirs ... FAIL
FAIL
Verifying:                 fifo ... FAIL
Verifying:              devices ... FAIL
Verifying:          combo-tests ...
   Sub-test:  xattrs + rsrc forks ... FAIL
   Sub-test:     lots of metadata ... FAIL
FAIL

This is a crying shame, since xar 1.5 and later do a great job. Hopefully people won’t pass judgment on xar based on the version shipped with Leopard.

On the other hand, rsync seems to have improved in Leopard:

------------------ rsync-apple ------------------
Verifying:    basic-permissions ... ok
Verifying:           timestamps ...
   Sub-test:    modification time ... ok
ok
Verifying:             symlinks ... ok
Verifying:    symlink-ownership ... ok
Verifying:            hardlinks ... ok
Verifying:       resource-forks ... ok
Verifying:         finder-flags ... ok
Verifying:         finder-locks ... FAIL
Verifying:        creation-date ... FAIL
Verifying:            bsd-flags ... ok
Verifying:       extended-attrs ...
   Sub-test:             on files ... ok
   Sub-test:       on directories ... ok
   Sub-test:          on symlinks ... FAIL
FAIL
Verifying: access-control-lists ...
   Sub-test:             on files ... ok
   Sub-test:              on dirs ... ok
ok
Verifying:                 fifo ... ok
Verifying:              devices ... ok
Verifying:          combo-tests ...
   Sub-test:  xattrs + rsrc forks ... ok
   Sub-test:     lots of metadata ... ok
ok

This is only in the svn version, folks, so don’t go downloading 1.4 and expecting greatness.

------------------ xar ------------------
Verifying:    basic-permissions ... ok
Verifying:           timestamps ...
   Sub-test:    modification time ... ok
ok
Verifying:             symlinks ... ok
Verifying:    symlink-ownership ... ok
Verifying:            hardlinks ... ok
Verifying:       resource-forks ... ok
Verifying:         finder-flags ... ok
Verifying:         finder-locks ... ok
Verifying:        creation-date ... ok
Verifying:            bsd-flags ... ok
Verifying:       extended-attrs ...
   Sub-test:             on files ... ok
   Sub-test:       on directories ... ok
   Sub-test:          on symlinks ... ok
ok
Verifying: access-control-lists ...
   Sub-test:             on files ... ok
   Sub-test:              on dirs ... ok
ok
Verifying:                 fifo ... ok
Verifying:              devices ... ok
Verifying:          combo-tests ...
   Sub-test:  xattrs + rsrc forks ... ok
   Sub-test:     lots of metadata ... ok
ok

Hmm, I guess that means it’s time to start thinking up some harder tests. Single-file backups? These are tricky on Mac because the finder puts the metadata for a whole folder into a single .DS_Store file. To do single-file backup right you need to be able to extract that one file’s metadata from the .DS_Store file, archive it, then reinsert it when you do a restore, all without disturbing the metadata of any other files in the folder. It can be done, but it’s not the easiest thing in the world.

------------------ xar ------------------
Verifying:    basic-permissions ... ok
Verifying:           timestamps ...
   Sub-test:    modification time ... ok
ok
Verifying:             symlinks ... ok
Verifying:    symlink-ownership ... ok
Verifying:            hardlinks ... ok
Verifying:       resource-forks ... ok
Verifying:         finder-flags ... ok
Verifying:         finder-locks ... ok
Verifying:        creation-date ... FAIL
Verifying:            bsd-flags ... ok
Verifying:       extended-attrs ...
   Sub-test:             on files ... ok
   Sub-test:       on directories ... ok
   Sub-test:          on symlinks ... ok
ok
Verifying: access-control-lists ...
   Sub-test:             on files ... ok
   Sub-test:              on dirs ... ok
ok
Verifying:                 fifo ... ok
Verifying:              devices ... ok
Verifying:          combo-tests ...
   Sub-test:  xattrs + rsrc forks ... ok
   Sub-test:     lots of metadata ... ok
ok

Now that’s one impressive result! They’ve got a bug in their tracker for the creation date issue, so hopefully it’ll be solved soon. There’s still an unfortunate issue with xar that prevents me from gushing uncontrollably about it — you can’t build a “xarpipe” to copy filesystem subtrees around. In other words, you always have to archive to a file, then unarchive from that file. This means that if you want to, say, transfer all your files from an old drive to a new drive you’d better have a third drive around, or your new drive had better be twice as big as the size of the files on your old one.

Get it while it’s hot!

Doing backups in OS X has always been a bit nerve-wracking. We’ve got resource forks, Finder flags, and various other odd metadata that don’t exist on other platforms. With the advent of Tiger, Apple told us that cp, tar, and rsync would finally be aware of our metadata and preserve it properly. And there was much rejoicing.

But then someone (or something?) named maurits actually tested these tools (and quite a few others), and the results weren’t pretty. This was a valuable wake-up call for many in the Mac community, but maurits never got around to releasing his test suite so we could do our own testing. Backup Bouncer is here to remedy this omission.

Backup Bouncer’s job is to keep the ugly backup tools out of the club. It’s here to show you exactly how bad (or, dare I say it, good?) your backup tools are. The way you use it is simple. First you create a couple of volumes:

bbouncer create-vol Src
bbouncer create-vol Dst

Next you create the test files on the source volume:

Now you use any old method you want to copy the files from /Volumes/Src to /Volumes/Dst. Once that’s done, you verify the copy and see how well (or poorly) it went. You can start by checking the “critical” properties — properties that you’ll almost always want preserved:

Next you can expand the scope to “important” properties — properties that power-users may find important or may be critical in the future:

Finally, you can see how the tool fares on the entire test suite:

As an example of how the results look, here are the results for ditto on the entire test suite:

Verifying:    basic-permissions ... ok
Verifying:           timestamps ...
   Sub-test:    modification time ... ok
ok
Verifying:             symlinks ... ok
Verifying:    symlink-ownership ... FAIL
Verifying:            hardlinks ... ok
Verifying:       resource-forks ... ok
Verifying:         finder-flags ... ok
Verifying:         finder-locks ... FAIL
Verifying:        creation-date ... FAIL
Verifying:            bsd-flags ... FAIL
Verifying:       extended-attrs ...
   Sub-test:             on files ... FAIL
   Sub-test:       on directories ... FAIL
   Sub-test:          on symlinks ... FAIL
FAIL
Verifying: access-control-lists ...
   Sub-test:             on files ... FAIL
   Sub-test:              on dirs ... FAIL
ok
Verifying:                 fifo ... FAIL
Verifying:              devices ... ok
Verifying:          combo-tests ...
   Sub-test:  xattrs + rsrc forks ... FAIL
   Sub-test:     lots of metadata ... FAIL
FAIL

Pretty bad, eh? On a more optimistic note, here are the results from Shirt Pocket Software’s SuperDuper!, one of the only tools to even come close to passing all tests:

Verifying:    basic-permissions ... ok
Verifying:           timestamps ...
   Sub-test:    modification time ... ok
ok
Verifying:             symlinks ... ok
Verifying:    symlink-ownership ... ok
Verifying:            hardlinks ... ok
Verifying:       resource-forks ... ok
Verifying:         finder-flags ... ok
Verifying:         finder-locks ... ok
Verifying:        creation-date ... ok
Verifying:            bsd-flags ... ok
Verifying:       extended-attrs ...
   Sub-test:             on files ... ok
   Sub-test:       on directories ... ok
   Sub-test:          on symlinks ... ok
ok
Verifying: access-control-lists ...
   Sub-test:             on files ... ok
   Sub-test:              on dirs ... ok
ok
Verifying:                 fifo ... FAIL
Verifying:              devices ... FAIL
Verifying:          combo-tests ...
   Sub-test:  xattrs + rsrc forks ... ok
   Sub-test:     lots of metadata ... ok
ok

Very nice!

Backup Bouncer also has a built-in set of “copiers”, to automate testing of command-line tools like cp, rsync, tar, asr, pax, ditto, and so on. You can run these tests using the “bbouncer copy” command:

You’ll get several pages of output, so I won’t show it all here. Finally, I’ve included an autopilot script that you can use to create some volumes, initialize the source files, run all the copiers, and verify the results.

You can get Backup Bouncer here.

Update: My ACL tests were broken. I’ve updated the output shown here to reflect the corrected tests.