October 5, 2011 / igrali

Live face detection on Windows Phone

detection, face, facelight, frame, live, Mango, phone, recognition, silverlight, videobrush, windows

Mango brings you the possibility to use your application to get the live image from camera. This has brought many new options to people interested in augmented reality and for applying all sorts of filters straightforward. I personally find the use for face recognition very exciting and interesting. Unfortunately, you cannot use OpenCV library (actually, the wrapper for .NET called EmguCV). Fortunately there are people who write software on Codeplex. In this article I will use FaceLight, a project for face recognition in Silverlight, created by René Schulte, and create a Windows Phone application that can detect faces on camera.

Preparing the FaceLight for Windows Phone

The example he used demonstrates the use of FaceLight in Silverlight for the Web. You need to do small modifications to the source code in order to make it work on the WP7. First of all, go here and download the latest version of FaceLight source code. Once you did that, open the solution in Visual Studio 2010. Go to the References and remove the reference to the System.Windows.Browser. After you do that, Visual Studio will show that there’s an error in MainPage.xaml.cs, but feel free to ignore it for now. You should have the following situation in the solution Explorer:

Delete the App.xaml, MainPage.xaml and the whole FaceLight.Web project. Change the build to Release. You should now have only one project in the solution, called FaceLight. Add a class called FaceLight.cs to the solution. This class will serve you for initializing all the needed objects. This code was the part of MainPage.xaml.cs, but I have used it for creating the FaceLight object which you’ll use from your project. Now, the class should look like this:

using System;
using System.Windows.Media;
using System.Windows.Media.Imaging;

namespace FaceLight
{
    public class FaceLight
    {

        IFilter erodeFilter;
        IFilter dilateFilter;
        HistogramMinMaxSegmentator segmentator;
        HistogramVisualizer histogramViz;

        public ColorRangeFilter SkinColorFilter { get; set; }

        public FaceLight()
        {
            try
            {
                SkinColorFilter = new ColorRangeFilter
                {
                    LowerThreshold = new YCbCrColor(0.10f, -0.15f, 0.05f),
                    UpperThreshold = new YCbCrColor(1.00f, 0.05f, 0.20f)
                };
                erodeFilter = new Erode5x5Filter();
                dilateFilter = new Dilate5x5Filter();
                segmentator = new HistogramMinMaxSegmentator();
                histogramViz = new HistogramVisualizer { Scale = 20 };
            }
            catch (Exception ex)
            {
                throw ex;
            }
        }

        public WriteableBitmap Process(WriteableBitmap bmpToProcess)
        {

            if (bmpToProcess == null)
                return null;

            var skin = SkinColorFilter.Process(bmpToProcess);
            var erode = erodeFilter.Process(skin);
            var dilate = dilateFilter.Process(erode);
            dilate = dilateFilter.Process(dilate);
            dilate = dilateFilter.Process(dilate);

            var histogram = Histogram.FromWriteabelBitmap(dilate);
            segmentator.Histogram = histogram;
            segmentator.ThresholdLuminance = histogram.Max * 0.1f;
            var foundSegments = segmentator.Process(dilate);

            histogramViz.Histogram = histogram;
            histogramViz.Visualize(dilate);

            var result = new WriteableBitmap(bmpToProcess.PixelWidth, bmpToProcess.PixelHeight);
            foreach (var foundSegment in foundSegments)
            {
                var c = foundSegment.Center;
                result.DrawEllipseCentered(c.X, c.Y, foundSegment.Width >> 1, foundSegment.Height >> 1, Colors.Green);
            }

            return result;
        }
    }
}

So, as I said, this is basically the original code from the FaceLight project. You will use it to create the FaceLight object, and then send the frame (WriteableBitmap) to the method Process which returns you another WriteableBitmap with the ellipse. This ellipse has the center in the recognized face. You overlay this WriteableBitmap with the video frames coming from the camera. That way you have the recognized face with a green ellipse around it. Have in mind that you can detect only one face with this project.

Build the solution. You should get the “Build succeeded” message and now the FaceLight.dll should be available at the Facelight/Bin/Release folder. Remember the location! :)

Using the Facelight in Windows Phone project for face recognition

Now, create a new Windows Phone project. Name it whatever you like, it’s not important. I named it FaceLive. Go to references, and add the reference to the FaceLight.dll you’ve created earlier. Add it to the MainPage.xaml.cs project, too. You will need the following namespaces, too:

using System.Threading;
using System.Windows;
using System.Windows.Input;
using System.Windows.Media.Imaging;
using Microsoft.Devices;
using Microsoft.Phone.Controls;

You’ll also need the WriteableBitmapEx reference. You can install it via NuGet in the Package Manager Console:

PM> Install-Package WriteableBitmapEx

If it succeeded, we’re done with removing and adding references for this article. :) Now, the fun part!

First of all, create the UI for the application. It’s nothing dramatic, just the VideoBrush object, through which you get the video frames, overlayed with the Image object to draw the ellipse. Make the supported orientations Landscape only, and Orientated LandscapeLeft:

<phone:PhoneApplicationPage
    x:Class="LiveFace.MainPage"
    xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
    ....
    SupportedOrientations="Landscape" Orientation="LandscapeLeft"
    ....>

Define the ContentPanel and delete the Application and Page titles:

<Grid x:Name="ContentPanel" Grid.Row="1" Margin="12,0,12,0">
            <Canvas x:Name="camera"
                    HorizontalAlignment="Stretch"
                    Tap="camera_Tap">
                <Canvas.Background>
                    <VideoBrush x:Name="cameraBrush" />
                </Canvas.Background>
            </Canvas>
            <Image x:Name="overlayRectangle"
                   HorizontalAlignment="Stretch"
                   Stretch="Uniform"/>
        </Grid>

As you probably noticed, I added the camera_Tap event handler to the tap event of the camera canvas. Now, in the MainPage.xaml.cs define the following:

PhotoCamera cam;
bool isFaceRecognizing = false;
bool pushFrames = false;

FaceLight.FaceLight faceLight;

private static ManualResetEvent pauseFramesEvent = new ManualResetEvent(true);

Thread frameThread;

Override the OnNavigatedTo and OnNavigatingFrom methods to make sure you initialize the PhotoCamera object which uses the primary camera and makes it the source for the cameraBrush, the VideoBrush object defined in xaml, and to dispose the cam object when it’s no longer needed:

protected override void OnNavigatedTo(System.Windows.Navigation.NavigationEventArgs e)
        {
            cam = new PhotoCamera(CameraType.Primary);
            cameraBrush.SetSource(cam);
            base.OnNavigatedTo(e);
        }

protected override void OnNavigatingFrom(System.Windows.Navigation.NavigatingCancelEventArgs e)
        {
            if (cam != null)
            {
                cam.Dispose();
            }
            base.OnNavigatingFrom(e);
        }

I mentioned the camera_Tap event handler. When you run the application, nothing happens. All you get are video frames from the camera. Unfortunately the emulator doesn’t do the job here, so you’ll need the real device, but it demonstrates what’s going on in the UI:

The black rectangle is moving around, and on real device you get the picture from camera. Now, the already mentioned camera_Tap event handler actually starts the process of recognizing faces:

private void camera_Tap(object sender, GestureEventArgs e)
        {
            if (!isFaceRecognizing)
            {
                InitFaceRecognizer();
                isFaceRecognizing = !isFaceRecognizing;
            }
            else
            {
                DestroyFaceRecognizer();
                isFaceRecognizing = !isFaceRecognizing;
            }
        }

If the process hasn’t started, it get’s initialized, and if it is, it gets destroyed. The methods for that are the following:

private void InitFaceRecognizer()
        {
            faceLight = new FaceLight.FaceLight();
            pushFrames = true;
            frameThread = new Thread(PushFrame);
            frameThread.Start();
        }

private void DestroyFaceRecognizer()
        {
            pushFrames = false;
        }

Basically, in the initializer, the program starts processing video frames (actually, starts a new thread with the method PushFrame. So, by now you must have realized that the PushFrame method is the most important, and gets the job done. Let’s analyze part by part. While the pushFrames bool is true, we make two WriteableBitmaps, and create an array of integers from the video buffer:

private void PushFrame()
        {

            while (pushFrames)
            {
                PhotoCamera phCam = (PhotoCamera)cam;
                int[] ARGBPx = new int[(int)cam.PreviewResolution.Width * (int)cam.PreviewResolution.Height];
                WriteableBitmap wb;
                WriteableBitmap wbBmp;
                pauseFramesEvent.WaitOne();

                phCam.GetPreviewBufferArgb32(ARGBPx);
                pauseFramesEvent.Reset();

Since it’s running on another thread, we need to use the Dispatcher.BeginInvoke to make changes to the UI. We copy the integer array to the wbBmp WriteableBitmap, which then contains the image from the video buffer, and give it to the method Process from the FaceLight library. It return another WriteableBitmap (the one with the green ellipse), and we save it in the wb WriteableBitmap. Finally, we draw the ellipse by setting the overlayRectangle source property to wb. For every frame, we get signaled and use the same method.

Deployment.Current.Dispatcher.BeginInvoke(delegate()
{
    wbBmp = new WriteableBitmap((int)cam.PreviewResolution.Width, (int)cam.PreviewResolution.Height);
    ARGBPx.CopyTo(wbBmp.Pixels, 0);
    wb = faceLight.Process(wbBmp);
    wb.Invalidate();
    overlayRectangle.Source = wb;
    pauseFramesEvent.Set();
});
}
}

When you test the project in daylight, it gets the job done almost always right. In the words of the author: “it runs in real time and works for most cases”. I hope you’ll find it useful in the Windows Phone world! :)

*EDIT: I have edited this article to make a clear distinction between “detect” and “recognize”. My article shows how to detect faces. Face recognizing would mean that the application has a database of faces which it uses for comparison to recognize who the person is. Thanks to Kristof Van De Voorde for leaving the comment!