The obvious approach for a pattern-based tracker like that in Shake, Flame and friends is to calculate the cross-correlation function between the pattern you're looking for and the search window. This gives you a measure of how well-matched the pattern is for every possible position in the window.
Any signal processing or engineering mathematics textbook will tell you more about correlation. The ever-excellent Paul Bourke has some examples working with 2D images here: http://local.wasp.uwa.edu.au/~pbourke/miscellaneous/correlate/
It's pretty slow as it is, but can be optimised in a whole bunch of ways, some of which are detailed here in a paper from ILM about the tracker used for Forest Gump and subsequently in Shake: http://www.idiom.com/~zilla/Work/nvisionInterface/nip.html
There are some implementation details in the Fxguide "Art of tracking article" too: http://www.fxguide.com/article209.html
There are open-source trackers out there in the computer vision world, such as in the OpenCV and VTK toolkits, but I think they tend to be more focused on tracking a lot of features at once for a camera solve rather than more precise single-feature tracking. They use similar techniques to optical flow or motion estimation - less precise but much faster than pattern-matching. I recall hearing that the Pixel Farm re-jigged PFTrack to use pattern tracking recently though, as part of the stereo update.