With the help of the good Dave Hyatt, I managed to track down the cause of the strange font behavior in the Cairo port. As it turns out, the problem was actually also present in the CG port, but had a slightly more benign failure mode.
You can read the gory details here.
You will also need the following two patches: