Before I start, I should address the inevitable “what about OAuth2″ comments. Yes, OAuth2 is the new hotness for Facebook authentication, but the Facebook Connect iPhone SDK isn’t going to get broken (intentionally!) any time soon. (I have that on authority from a friend inside Facebook, BTW.) If you’ve got the time to replicate the functionality of FBConnect for the new world order I salute you, but otherwise this code may come in handy and will continue to function until a full-featured replacement for FBConnect arrives.

With a session proxy (say at http://your-server.example.com/fbproxy), this is how the client establishes a session:

• Phone:
  – GET http://your-server.example.com/fbproxy?auth_token=A_TOKEN&…
• To respond, your server relays the results of this POST:
  – POST: http://api.facebook/com/restserver.php
  – POST body: auth_token=A_TOKEN&sig=A_SIGNATURE&…

A_SIGNATURE is generated using your FB app secret, so Facebook knows the request is legit. Facebook returns a session key & secret that can be used to make Facebook API calls from the phone. If you’re wondering (like I did) how this is any more secure than shipping your app secret directly, you should understand that a session key is much less powerful than an app secret. The app secret is basically the root password for your app, while a session key gets you an unprivileged user account.

With HexaLex I’ve used Google App Engine to host most of the backend service, so I ended up adapting some sample code I found online to do the job. (minifb is particularly readable.) I didn’t want to pull in a bunch of other dependencies, so this code is self-sufficient. I’m using Google’s webapp framework so that’s what’s used in the code, but it should be easy to adapt to whatever framework you favor. Without further ado, here’s the code:

"""
To use this, write in your Facebook API key and secret below and then
add FacebookSessionProxy to your list of url handlers:
    application = webapp.WSGIApplication(
        [('/fbproxy', FacebookSessionProxy),
         ...])
 
Then in your iPhone code use something like
session = [FBSession sessionForApplication:myApiKey 
             getSessionProxy:@"http://yourApp.appspot.com/fbproxy"
             delegate:self];
"""
import hashlib
import httplib
import urllib
from google.appengine.ext import webapp
 
# This lambda prevents the secret from appearing in cgitb backtraces
FB_API_SECRET = lambda:"YOUR_API_SECRET"
FB_API_KEY = "YOUR_API_KEY"
 
FB_API_HOST="api.facebook.com"
FB_API_PATH="/restserver.php"
 
def facebook_signature(paramsDict):
    """Compute the signature of a Facebook request"""
    sorted = paramsDict.items()
    sorted.sort()
 
    trace = ''.join(["%s=%s" % x for x in sorted])
    trace += FB_API_SECRET()
 
    md5 = hashlib.md5()
    md5.update(trace)
    return md5.hexdigest()
 
def get_fb_session(auth_token):
    """Request a Facebook session using the given auth_token"""
    params={
            "api_key":FB_API_KEY,
            "v":"1.0",
            "auth_token":auth_token,
            "generate_session_secret":"1",
            "method":"auth.getSession",
    }
    params["sig"] = facebook_signature(params)
 
    encoded_params = urllib.urlencode(params)
    headers = {
            "Content-type":"application/x-www-form-urlencoded",
    }
 
    conn = httplib.HTTPConnection(FB_API_HOST)
    conn.request("POST", FB_API_PATH, encoded_params, headers)
    return conn.getresponse()
 
class FacebookSessionProxy(webapp.RequestHandler):
    def get(self):
        response = self.response
        auth_token = self.request.get('auth_token')
        if len(auth_token) == 0:
            response.set_status(httplib.BAD_REQUEST)
            response.out.write("Facebook login error: no auth_token given.")
            return
        fbResponse = get_fb_session(auth_token)
        response.out.write(fbResponse.read())
        response.set_status(fbResponse.status, fbResponse.reason)
        response.headers['Content-Type'] = fbResponse.getheader('content-type')
 
# The End

Update: I fixed a bug in the call to get_fb_session. Thanks to Justin Williams for reporting it.

Slides

Audio

Video

Or you can just search the iTunes Store for “Backup Bouncer”. Surprisingly enough, my talk is the only hit.

It was nice to get a chance to talk about my work, but a bit strange to give a talk over Skype. I was expecting a video feed from the other end but there wasn’t any, so for all I know I was giving the talk to an empty room. This had the beneficial side effect of totally eliminating any stage fright I might have felt, so maybe it was a net win.

Skype doesn’t support screen sharing and webcam video simultaneously, so the poor attendees were stuck looking at my (very static) slides while listening to my disembodied voice. Furthermore, judging by the stuff on iTunes the sound quality of my bluetooth headset let a bit to be desired. (Geez, you sink a whole $5 into a bluetooth headset and you get junk! Where is the love?)

Some of you have probably noticed that I’ve been neglecting some of my other projects and complaining about being busy a lot lately. Well here’s what I’ve been working on. I’m pleased to announce HexaLex, a new angle on crossword games! The first platform for HexaLex is iPhone and iPod Touch, but I’m planning to expand to Facebook in the near future.

HexaLex as it looks in-game

If you want to get a quick overview of the game and a list of features go ahead and visit the HexaLex.com website. If you want to read some great 5-star reviews or buy the game just pop on over to the App Store. Continue reading for a little more detail on the core concept of the game.

HexaLex is an idea I came up with a little over a year ago as a grad student. The core gameplay is a lot like Scrabble, Words With Friends, or Lexulous, so if you’ve played one of those games you’ll get the idea right away. You use tiles to build words on a game board. But HexaLex has a big difference — it’s played with hexagonal tiles. Looks interesting, right?

Playing a crossword game with hexagonal tiles sounds like a simple idea but it actually requires a bit of creative thinking to make it playable and fun. (This is probably why it hasn’t been done a million times already!) You see, when two words cross with square tiles they only interact at one tile — the tile of intersection. With hexagonal tiles, that’s not the case. Hex tiles cause interactions at the tile of intersection and its neighbors! This means that playing one word across another word also forms two new two-letter words. This is sort of hard to describe, but easy to see as a picture (like the one below).

Crossing START with CRAZY forms RT and RZ

Traditionally, in crossword games you have to make every word on the board valid. If you just take the standard rules and try to use them with hexagonal tiles you’ll quickly discover that the game is unplayably hard. It’s just frustrating to have to work out two valid two-letter words for each and every crossing play.

The way I solved this in HexaLex was to introduce the idea of junk words. A junk word is a two-letter word that’s not a valid word. So XR, TQ, and ZA are examples of junk words. (Oops, ZA is actually a valid word, at least in the strange world of crossword games.) In HexaLex you are allowed to play junk words subject to a few restrictions:

The main menu features a sliding word animation

With junk words, a simple play of one word crossing another is always possible provided the two words share a letter. The game is playable once again and balance is restored!

So now you know that when you look at a HexaLex game it’s OK that some of the two-letter words are bogus, but all of the longer words have to be legit.

Junk words aren’t the only break from Scrabble tradition in HexaLex but I’ll leave the rest for another post.

HexaLex is available now in the App Store. Give it a shot! Initial App Store reviews have been overwhelmingly positive, with fourteen unanimous 5-star ratings and reviews so far. If you enjoy HexaLex please let me know! You can also become a HexaLex fan on Facebook and/or follow HexaLexGame on Twitter. Also, please help spread the word by clicking the digg button!

To make matters worse, the system’s top-secret non-configurable algorithm for resolving conflicts appears to work backwards from what users might hope for. The Xcode plugin lives inside an Apple-supplied app bundle in a system directory. It’s installed automatically if you choose to install Xcode itself, so it can easily up on your system without you even knowing it. Compare this to QLCC. You, the user, have actively downloaded it and installed it for the express purpose of handling source code files in the Quick Look system. You can easily remove it if you decide you don’t like it anymore. Which plugin do you think should win if they conflict? Apple thinks different. The system always picks the Xcode plugin over QLCC.

But come on, if you’re going to have a system like this there ought to be a way of saying “I want plugin X to handle UTI Y.” This could be a nice shiny user interface somewhere or a down-n-dirty terminal command, but it needs to exist. I sent a bug report to Apple saying as much (rdar://7240036, mirrored here). I described my problem and made it clear that the solution ought to be a preference system for Quick Look. At the very least, I said, user-installed plugins should rule supreme.

Apple engineering’s response? You can register for more specific UTIs than public.source-code. Nothing about why they don’t want to have a preference system or let user-installed plugins override others. Just “register for more specific UTIs.” The idiocy of this suggestion is staggering. The whole purpose of the hierarchical nature of the UTI system is to allow folks like me to say “I handle source code” without having to list every type of source code. Highlight, the library that QLCC is based on, can colorize about 140 programming languages. Apple engineering, with a straight face, is telling me that I should be registering for 140 UTIs because Xcode decided to register for the one I actually need.

It gets worse. Even if I decided to take Apple’s advice, it’s impossible to say what those 140 UTIs should actually be, because there’s no central authority for registering UTIs. (This is another glaring flaw with the UTI system as implemented by Apple.) UTIs like org.ocaml.ocaml-source are generally defined and bound to file extensions like .ml by applications or other entities like Quick Look plugins. So what happens if you install one app that binds .ml to org.ocaml.ocaml-source and another that binds it to fr.inria.ocaml-source? Your guess is as good as mine. That’s another bit of non-configurable secret sauce. But I think it’s safe to say that only one will win.

So let’s go back to my hypothetical 140-UTI plugin. Imagine the confusion and headaches when a user’s .ml files stop being highlighted just because she happened to download some random app that bound .ml to fr.inria.ocaml-source and my plugin has registered for org.ocaml.ocaml-source. Can you imagine trying to solve these problems for random users of 140 programming languages, of which you actively use maybe 10?

But really, this is just an example of brush-off via bug report. Despite my slagging I do believe the engineers at Apple are generally quite talented and bright. They must know that the problem exists, but for whatever reason they’ve decided it’s not worth solving. Furthermore, they’ve decided it’s not worth telling some no-name hacker why they think it’s not worth solving. But the most insulting thing is that they didn’t even bother to mark my issue as a duplicate of some other issue requesting a prefs system for Quick Look. (You’d better believe that one exists.)

“Go register for more specific UTIs, kid, yer bothering me.”

¹ If you’re unfamiliar with the UTI system, John Siracusa’s encyclopedic review of Tiger has a detailed description.

For history’s sake, I’m observing this on Snow Leopard 10.6.1, Xcode 3.2, Instruments 2.0.

Continuing on my “update side projects for Snow Leopard” theme, I’ve just rolled a new release of QLColorCode. I added support for a few new languages, including Scala, which is a personal fave. I also fixed a bug or two and finally upgraded the bundled version of Highlight from 2.6.6 to 2.12, which was long overdue.

On the ‘meta’ side, I moved the project’s source code from google code’s subversion to github. I’ve been using git a lot lately so I expect I’ll do the same for my other projects as well. Once you’ve gotten used to firing off branches for every little thing that crosses your mind and seamlessly switching between them at a moment’s notice it becomes hard to work in the SVN mindset again.

Unfortunately, QLCC has hit a bit of a roadblock with Snow Leopard. More specifically, it’s not SnoLep that’s giving me a headache, it’s Xcode 3.2. With the new Xcode Apple decided to include their own QL plugin for source code. That’s great, I say, more choice is good! QLCC will render everything that Xcode renders plus OCaml files, Scala files, and all sorts of other interesting languages that Xcode can’t handle. I’m not worried about QLCC losing in a fair fight. But no, Apple doesn’t fight fair.

It turns out that Quick Look always picks the Xcode plugin when rendering source code files (on my machine, at least). I don’t think the Xcode plugin gets special treatment, it just happens to win in the algorithm QL uses to resolve plugin conflicts. What is that algorithm? Nobody outside of Apple knows. It could be a coin-flip for all we know. Heck, after a reboot it’s possible QLCC will always win! In any case, when there are two plugins that both want to render files of type public.source-code it seems like maybe letting the user pick one would be a good idea, right? Apple doesn’t think so. No, they don’t want users to worry their pretty little heads about such matters. Don’t worry about the algorithm, don’t worry about making choices, it’ll all be juuust fine. Have a lollipop.

Except it’s not all fine. Because let’s say a user decides that they prefer QLCC over the Xcode plugin. Their only option for getting QLCC consistently and predictably is to somehow disable Xcode’s plugin. They could remove it, rename it, or move it somewhere that quicklookd won’t find it. But who wants to do something like that to an Apple-supplied file? Nobody. Even worse, Xcode’s application bundle is cryptographically signed. I believe that disabling the plugin by any of those means will invalidate the signature, though I haven’t confirmed this as fact.

Update: After further research, it appears that disabling the plugin will not invalidate the signature on Xcode itself, so there shouldn’t be any terrible consequences if you choose to do so. The plugin is located at: /Developer/Applications/Xcode.app/Contents/Library/QuickLook/SourceCode.qlgenerator

So anyhow, QLColorCode 2.0 is out. It works with Snow Leopard. If you have Xcode 3.2 installed you’ll probably never get to see it do its thing, but maybe one or two of you will still get something out of it.

For what it’s worth, quick look still sends .plist files to QLCC. Xcode’s plugin won’t touch those things.

All is not quite well on the Snow Leopard front, however. The rsync that Apple ships (at least the one in 10.6.1) has a bug. When copying a FIFO it attempts to copy the contents of the FIFO. If you know what a FIFO is, you know why this is bad. So rsync hangs in the 90-fifo test of the test suite. When running from a terminal you can work around this by just hitting Control-C, which will cause the copy to stop and the other copiers to be tested. If you want to avoid the hang you can rename tests.d/90-fifo.test to tests.d/90-fifo.test.disabled or whatever floats your boat. I haven’t done that already because, well, a hang is arguably a valid test result!

Oh, and I also fixed a bug with the BSD flags test. If you care about that stuff you should re-test.

One other interesting tidbit — I changed the “lots of metadata” test to lock the file. This trips up some copiers that duplicate the “locked” status before setting up other metadata, causing the other metadata to be lost.

Much to my shame, I had an e-mail upheaval and I can’t find the names of the individuals who reported the BSD flags bug and the “early locking” phenomenon. If that was you, please drop me a message and I’ll give you the credit you deserve.

UPDATE: It was Rob Kennedy who reported the BSD flags bug and the “early locking” phenomenon. I hadn’t thought to search through my blog comments. Silly me! Thanks a lot for those tips, Rob!

Before you get your hopes up, please understand that this contact was made back in the early summer and nothing much has come from it. Hopefully a few “votes” for the issue will get the ball rolling again.

Last week I randomly decided to google around and see if anybody else had found a solution to the problem. Well, the results of the search were pretty interesting. The problem was ubiquitous — tons of folks with PCR controllers built around the same time as mine reported it on forums and review sites. (This was not the case when I first discovered the problem, by the way.) Some people reported that opening the keyboard and rubbing the key contacts with a stiff pencil eraser brought them back to life, which got me thinking. These things are just electrical contacts, right? Why not put conductive paint on the contacts? That should get them working again for good!

So I did a goog for “conductive paint” and one of the first links that came up was this one. And hey, can you believe it? They sell a “Rubber Keypad Repair Kit”! (MG Chemicals, cat. #8339.) Hallelujah, exactly what I wanted! It’s actually designed for repairing remote controls, but midi controllers work on the same principle. I decided to order one and gave it a shot. (BTW, my experience with Action Electronics was very good.)

How did it turn out? Well, it was good but not great. The kit did, in fact, manage to revive almost all of my keys. Why didn’t it work for all of them? Well, each key actually has two contacts underneath, and a bit of experimentation revealed that the time interval between closing the two contacts is quite important for the keyboard performance. (I assume this is how they sense the velocity with which you struck the key.) If they close in the wrong order, for example, no note is triggered. I think that the added thickness of the paint (which didn’t always go on particularly smoothly) caused timing problems with some of the keys, perhaps even causing the contacts to close in the wrong order. I tried sanding off the paint and repainting the bad keys, but it never quite worked for all the keys.

I tried to live with this for a while — it was, after all, a big improvement over the keyboard’s previous condition. But the fact is, there are times in any musician’s life when only D# will do, and using D or E as a workaround just doesn’t cut it. Finally, I decided that it couldn’t hurt to call Roland and try talking to a live person. After some bouncing around in their phone system (boy do they have awful muzak when you’re on hold!) they actually gave me approval to send in my keyboard! This despite it being many years out of warrantee and my admission that I’d tried to fix the problem myself!

I was a little bit afraid that their technician would investigate the problem, find the contacts covered with conductive paint, and stamp DENIED on my RMA ticket with an evil laugh, but no such thing happened. I suspect they just swapped out the whole keyboard assembly without ever uncovering my handiwork — why bother removing all the keys when this is a known defect? So after 3 days I had my PCR-30 back in fully working condition for no cost other than the time and gas required to drop the thing off at the nearby Roland factory and pick it up again. Kudos to Roland for making good after all these years!

So anyhow, the moral of the story is that talking to people by phone works better than e-mail, at least if those people are Roland employees. Or maybe it’s that conductive paint doesn’t work as well as you might hope.

First, to access a shared printer on another machine, you have to turn on print sharing on your machine. This defies all logic, but there you go.

Second, for some utterly inscrutable reason Apple decided to *dis*able the CUPS browsing protocol, leaving only Bonjour browsing working. ???? Stupid. Stupid. Stupid. It’s as if they said “let’s make sure Mac users will have lots of trouble printing to Linux servers, that’ll improve our customer satisfaction!” To fix this lameness, do the following:

Combining these two tips, the shared printer finally appeared in the “Default” pane of the new printer dialog and printing to it just worked. Hopefully Apple makes this more sensible in Snow Leopard…

Here’s how you test Time Machine with Backup Bouncer:

$ mkdir ~/bb
# We need to trick BB into thinking this is a volume it created
$ touch ~/bb/bbouncer-vol
$ sudo bbouncer create ~/bb
# Now do a Time Machine backup and wait till it's done
$ mv ~/bb ~/bb-orig
# Now restore the ~/bb directory from Time Machine.  
# Try not to catch SPACE MADNESS!!!
$ bbouncer verify -d ~/bb-orig ~/bb

On my Leopard 10.5.3 machine here’s what I get with BB 0.1.3:

Verifying:    basic-permissions ... ok (Critical)
Verifying:           timestamps ... ok (Critical)
Verifying:             symlinks ... ok (Critical)
Verifying:    symlink-ownership ... FAIL 
Verifying:            hardlinks ... FAIL (Important)
Verifying:       resource-forks ... 
   Sub-test:             on files ... ok (Critical)
   Sub-test:  on hardlinked files ... FAIL (Important)
Verifying:         finder-flags ... ok (Critical)
Verifying:         finder-locks ... ok 
Verifying:        creation-date ... ok 
Verifying:            bsd-flags ... FAIL 
Verifying:       extended-attrs ... 
   Sub-test:             on files ... ok (Important)
   Sub-test:       on directories ... ok (Important)
   Sub-test:          on symlinks ... ok 
Verifying: access-control-lists ... 
   Sub-test:             on files ... ok (Important)
   Sub-test:              on dirs ... ok (Important)
Verifying:                 fifo ... FAIL 
Verifying:              devices ... FAIL 
Verifying:          combo-tests ... 
   Sub-test:  xattrs + rsrc forks ... ok 
   Sub-test:     lots of metadata ... ok 

The only annoyance is that hardlinks aren’t preserved, but that’s not surprising considering the way hardlinks are used internally by TM to prevent wasted space. (I should point out that the hardlink failure here is not the same as the Apple rsync one I mentioned earlier today, and it’s not nearly as bad.) These results are not perfect but I would have no problem recommending TM backups to any less-than-hardcore user.

------------------ rsync-apple ------------------
Verifying:    basic-permissions ... ok (Critical)
Verifying:           timestamps ... ok (Critical)
Verifying:             symlinks ... ok (Critical)
Verifying:    symlink-ownership ... ok 
Verifying:            hardlinks ... ok (Important)
Verifying:       resource-forks ... 
   Sub-test:             on files ... ok (Critical)
   Sub-test:  on hardlinked files ... FAIL (Important)
Verifying:         finder-flags ... ok (Critical)
Verifying:         finder-locks ... FAIL 
Verifying:        creation-date ... FAIL 
Verifying:            bsd-flags ... ok 
Verifying:       extended-attrs ... 
   Sub-test:             on files ... ok (Important)
   Sub-test:       on directories ... ok (Important)
   Sub-test:          on symlinks ... FAIL 
Verifying: access-control-lists ... 
   Sub-test:             on files ... ok (Important)
   Sub-test:              on dirs ... ok (Important)
Verifying:                 fifo ... ok 
Verifying:              devices ... ok 
Verifying:          combo-tests ... 
   Sub-test:  xattrs + rsrc forks ... ok 
   Sub-test:     lots of metadata ... ok 

If you add a resource fork to a file that’s hardlinked to another file then only one of the files gets copied. This, no doubt, is due to the hackery involved to make HFS+ look like it has a real hardlinks when it actually doesn’t.

Some of the changes in BB 0.1.3 were a bit invasive and Bash is not the greatest of programming languages, so please keep your eyes open for bugs. If you see something in your results that doesn’t make sense just let me know.

So I wasn’t too surprised when a user forwarded me a somewhat exasperated-sounding note from Shirt Pocket software explaining why SD doesn’t pass BB’s FIFO and device file tests. I won’t paste the note in here, since I feel that would be poor form, but I will pass along their rationale for not copying these files. They feel that FIFOs and device files are normally system-created files that are not meant to be preserved across reboots or backups. For that reason, SD doesn’t back up these filesystem object types at all.

One can understand how Shirt Pocket arrived at this policy. Writing a good system-level backup tool is more subtle than it might appear. There are actually parts of the filesystem that you don’t want to back up, like caches, VM files, and so forth. You don’t want to back up the device files under /dev, since those get created dynamically depending on what devices are attached. In short, there are some parts of the filesystem that are tied to the state of the running system, and these should not normally be copied in a backup.

The problem with Shirt Pocket’s policy is that although it is certainly true that the system-created FIFOs and device files should not be copied in a system-level backup, there are legitimate reasons a user might create FIFOs or device files, and there’s every reason to preserve those user-created objects during a backup. Don’t get me wrong — this is obscure stuff that only advanced UNIX hackers would do, but hey, advanced UNIX hackers need reliable backups too! SD’s policy is, IMHO, overly broad. I would be much more comfortable if SD excluded filesystem locations rather than entire classes of filesystem objects.

The Shirt Pocket letter also accused BB of being “rather misleading to most” on this issue because it makes people worry about something that won’t cause them any problems. Although I vehemently disagree with the “misleading” characterization, I agree that not every BB test is equally important in practical terms. Most end-users will absolutely not care that tool X doesn’t backup FIFOs. This is why BB has always had a prioritization feature. Quoting from the usage string of bbouncer:

-T <set>     Use the given set of tests.  <set> can be either 
             "critical", for tests whose failure will cause problems
             for average users, or "important", for tests whose
             failure may not cause problems for many users, but which
             may be important to power-users or may qualify as 
             critical in the future.

The FIFO/device file tests are not included in either the “critical” or “important” set of tests. So I’m saying (and have always said) that most users, even most power users, won’t care about them. So why test for them at all? Consider the situation before Maurits’ blogged on backups and BB existed. Most people, myself included, didn’t even know what the full set of OS X filesystem object types and metadata was. Maurits filled us in on what existed, but knowing how to test for preservation was still a black art. BB was meant to democratize that testing, but also to act as an exhaustive test set. (I don’t claim to have achieved even close to 100% coverage, but that’s the goal.) So it’s important to me that BB include tests for any metadatum or filesystem object that can conceivably have a reason to be backed up.

Having said that, I’ll concede there’s a usability bug here. The -T option is unlikely to be discovered by casual users, and they’re the ones who need it most! It’s not used by autopilot and it’s not even documented in the README. So I think I’ll be making the following changes:

• I’ll document -T more pervasively
• I’ll have the priority of a test appear in BB’s output
• I’ll have BB default to running in “important” mode, changing the exhaustive mode to an opt-in option

Hopefully this will help prevent confusion for casual users and ease the support burden of tool developers while maintaining BB’s goals of providing easy backup tool validation and exhaustive test coverage.

I think the grownup meal is called a Sad Meal. It’s sad because it doesn’t have a toy.

Mind you, she was completely serious about this statement and delivered it with a totally straight face. Had my mouth been full of soda it would have come shooting out of my nose for sure…

Get it at the goog.