I rewrote the Non Rigid Face Tracking code by Jason Saragih (http://jsaragih.org/), which is published in the book “Mastering OpenCV with Practical Computer Vision Projects”. Copyright Packt Publishing 2012. (http://www.packtpub.com/cool-projects-with-opencv/book)
Mainly because I want to understand how it works, and partly because I wanted to clean up the code (which is arguably kind of messy).
Basically the algorithm goes like this:
1.Apply Active Shape Model, which consists of:
- Performing General Procrustes Analysis to align all the landmark points obtained from MUCT Face database in order to find a canonical shape. With this we obtain the rigid basis transformation matrix.
- Apply PCA to the points, in order to reduce features dimensionality, thus only k eigenvectors are preserved.
- Find and preserve 3 standard deviations of the found parameters, thus preserving 99.7% of the modes.
2. Learn the correlative patch models:
- Basically for each landmark points, we try to learn the patch from the sample images, searching for best response map for each patches.
- This is done by using stochastic gradient descent as the optimisation function, thus randomly choosing n-images to learn each patch.
Here’s a visualised sample of learned patch models, from about 1000 random samples:
3. Utilizing Viola Jones / Haar Cascade Classifier to detect face position:
- During training period, we apply the cascade classifier on the sample images, in order to obtain average offset of the landmark points
4. Tracking face
- With cascade classifier, we obtain the approximate bounding box where the faces are, and we could approximately overlay the shape model on top of it
- Refine the location of the points, by utilising the patch model. Calculate response for each corresponding patches for a given search window, and use the positions that gives the best response.
And here’s a sample result of applying them onto the casts from GiantBomb.com:
And here’s with predefined connections between the points:
And lastly, with triangle subdivisions:
You can get my example codes here: https://github.com/subokita/Sandbox/tree/master/NonRigidFaceTracking
It requires OpenCV, Intel TBB and Boost libraries. The face database can be downloaded from http://www.milbo.org/muct/